Display apparatus, method for controlling display apparatus, and program

ABSTRACT

A head mounted display is worn on the body of a user before use and includes an image display unit that transmits an outside scene and displays an image in a visually recognizable manner along with the outside scene and a camera that performs image capturing in directions of sight lines of the user. A control section of the head mounted display includes an image generation section that generates a display image from a captured image from the camera and an image display control section that causes the image display unit to display the display image generated by the image generation section.

BACKGROUND

1. Technical Field

The present invention relates to a display apparatus, a method forcontrolling the display apparatus, and a program.

2. Related Art

There is a known wearable display apparatus that displays an imagesuperimposed on an outside scene (see JP-A-2014-56217, for example). Theapparatus described in JP-A-2014-56217 has a configuration in which anoutside scene and characters are superimposed on each other whendisplayed and displays part of the characters or words with the fontsize, the color, or any other attribute thereof so changed thatcharacters in part of a sentence, for example, are readily identifiable.

JP-A-2014-56217 describes a display method in a case where a user whouses the wearable display apparatus views characters displayed by thedisplay apparatus or an outside scene. There has been, however, noproposal on a display method that takes into account a case where theuser views characters along with an outside scene.

SUMMARY

An advantage of some aspects of the invention is to provide a displayapparatus, a method for controlling the display apparatus, and a programthat allow an outside scene visible to a user to be effectively combinedwith a displayed content.

A display apparatus according to an aspect of the invention is a displayapparatus worn on a body of a user before use, the apparatus including adisplay section that transmits an outside scene and displays an image ina visually recognizable manner along with the outside scene, an imagingsection that performs image capturing along directions of sight lines ofthe user, an image generation section that generates a display imagefrom a captured image from the imaging section, and an image displaycontrol section that causes the display section to display the displayimage generated by the image generation section.

According to the aspect of the invention, a display image based on acaptured image captured along the directions of the sight lines of theuser can be so displayed that the display image is superimposed on atarget object visible as the outside scene. A novel method for using animage displayed by the display apparatus can therefore be provided inthe form of an effective combination of an outside scene outside thedisplay apparatus and a displayed content, for example, by enhancing theway the user views the outside scene based on a captured image generatedby capturing the outside scene or presenting another way of viewing theoutside scene.

The display apparatus according to the aspect of the invention describedabove may further include a detection section that detects a state inwhich the display apparatus is used, and the image display controlsection may adjust a form in accordance with which the display sectiondisplays the display image in correspondence with the state in which thedisplay apparatus is used and which is detected by the detectionsection.

According to the aspect of the invention, the display size, the displayposition, and other aspects of an image displayed in the display sectioncan be adjusted in correspondence with the environment in which thedisplay apparatus is used and the way in which the user uses the displayapparatus, whereby the display operation can be performed in accordancewith the way in which the user visually recognizes an outside scene.

In the display apparatus according to the aspect of the inventiondescribed above, the detection section may detect the position of thedisplay apparatus, and the image display control section may adjust theform in accordance with which the display image is displayed incorrespondence with the position of the display apparatus detected bythe detection section.

According to the aspect of the invention, the display operation can beperformed based on the position of the display apparatus in accordancewith the way in which the user visually recognizes an outside scene.

In the display apparatus according to the aspect of the inventiondescribed above, the image display control section may adjust at leastone of a display position and a display size of the display image insuch a way that the display image is separate from the center of thedirections of sight lines of the user in correspondence with theposition of the display apparatus detected by the detection section.

According to the aspect of the invention, an image captured with theimaging section can be so displayed that the image does not bother theuser who views an outside scene.

The display apparatus according to the aspect of the invention describedabove may further include an action detection section that detectsaction of the user based on a captured image from the imaging section,and the image display control section may adjust the form in accordancewith which the display image is displayed in correspondence with theaction of the user detected by the action detection section.

According to the aspect of the invention, by allowing the user toperform action an image of which can be captured with the imagingsection, the form in accordance with which the display image isdisplayed can be adjusted.

The display apparatus according to the aspect of the invention describedabove may further include a voice detection section that detects voice,and the image display control section may adjust the form in accordancewith which the display image is displayed in correspondence with aresult of the voice detection from the voice detection section.

According to the aspect of the invention, by allowing the user to speak,the form in accordance with which the display image is displayed can beadjusted.

In the display apparatus according to the aspect of the inventiondescribed above, the image generation section may generate the displayimage by extracting and processing part of a captured image from theimaging section.

According to the aspect of the invention, since a captured imagecaptured along the directions of the sight lines of the user is used toperform the display operation, the user is allowed to view an outsidescene and an image generated by processing part of the outside scene.

The display apparatus according to the aspect of the invention describedabove may further include an action detection section that detectsaction of the user based on a captured image from the imaging section,and the image generation section may generate the display image byextracting, from the captured image from the imaging section, an areaidentified based on the action of the user detected by the actiondetection section and processing the extracted area.

According to the aspect of the invention, by allowing the user toperform action an image of which can be captured with the imagingsection, the area extracted from the captured image can be determined.

In the display apparatus according to the aspect of the inventiondescribed above, the image generation section may generate the displayimage by cutting out and enlarging part of a captured image from theimaging section.

According to the aspect of the invention, since part of a captured imagecaptured along the directions of the sight lines of the user is cut outand enlarged before displayed, the user can view an outside scene and aportion thereof along the directions of the sight lines as an enlargedimage.

In the display apparatus according to the aspect of the inventiondescribed above, the image generation section may generate the displayimage by cutting out an area that satisfies a preset condition in acaptured image from the imaging section.

According to the aspect of the invention, a captured image thatsatisfies the preset condition can be displayed. For example, when acaptured image that satisfies a preference or an interest of the user iscaptured, the display section can display the captured image and show itto the user.

The display apparatus according to the aspect of the invention describedabove may further include a target detection section that identifiesdirections corresponding to the directions of the sight lines of theuser and detects a target object present in the directions of the sightlines of the user from a captured image from the imaging section, andthe image generation section may generate the display image byextracting, from the captured image from the imaging section, an areacontaining an image of the target object detected by the targetdetection section and processing the extracted area.

According to the aspect of the invention, when a target object ispresent in the directions of the sight lines of the user, an image ofthe target object can be presented to the user.

In the display apparatus according to the aspect of the inventiondescribed above, the target detection section may detect the directionsof the sight lines of the user and detects the target object present inthe detected directions of the sight lines of the user.

According to the aspect of the invention, a portion corresponding to thedirections of the sight lines of the user in a captured image can bedisplayed as the display image.

In the display apparatus according to the aspect of the inventiondescribed above, the target detection section may detect the directionof the sight line of an eye corresponding to a dominant eye of both eyesof the user.

According to the aspect of the invention, a portion corresponding to thedirection of the sight line of the dominant eye of the user in acaptured image can be displayed as the display image.

In the display apparatus according to the aspect of the inventiondescribed above, when the action detection section detects predeterminedaction of the user, the target detection section identifies thedirections corresponding to the directions of the sight lines of theuser based on the predetermined action.

According to the aspect of the invention, a portion to be extracted asthe display image from a captured image can be specified by the actionof the user.

In the display apparatus according to the aspect of the inventiondescribed above, when the action detection section detects action of atleast one of limbs of the user, the target detection section mayidentify the directions corresponding to the directions of the sightlines of the user based on the detected action.

According to the aspect of the invention, by allowing the user to moveany of the limbs, a portion to be extracted as the display image from acaptured image can be specified.

The display apparatus according to the aspect of the invention describedabove may further include a storage section that stores the amount offeature of an image of the target object, and the target detectionsection may detect the target object by detecting an image that conformsto the amount of feature stored in the storage section from a capturedimage from the imaging section.

According to the aspect of the invention, a target object can be readilydetected based on a captured image.

In the display apparatus according to the aspect of the inventiondescribed above, the amount of feature stored in the storage section maycontain one of the amount of feature of an image of a specific targetobject and the amount of feature for general purposes that correspondsto a plurality of types of target objects similar to each other.

According to the aspect of the invention, the process of detecting aspecific target object or the process of detecting any of a plurality oftarget objects similar to each other can be carried out based on thestored amount of feature.

The display apparatus according to the aspect of the invention describedabove may further include a distance detection section that detects adistance between the target object detected by the target detectionsection and the user, and the image display control section may generatethe display image by enlarging part of a captured image from the imagingsection in accordance with an enlargement factor according to thedistance detected by the distance detection section.

According to the aspect of the invention, an image of the target objectcan be enlarged in accordance with the enlargement factor correspondingto the distance to the target object present in the directions of thesight lines of the user, and the enlarged image can be presented to theuser.

In the display apparatus according to the aspect of the inventiondescribed above, the image generation section may generate the displayimage by acquiring a plurality of captured images captured by theimaging section at different points of time and specifying an area thatsatisfies a preset condition in the plurality of captured images.

According to the aspect of the invention, a complicated condition can beset on the plurality of captured images, and images based on capturedimages that satisfy the condition can be presented to the user.

In the display apparatus according to the aspect of the inventiondescribed above, the image generation section may generate the displayimage by acquiring a plurality of captured images captured by theimaging section at different points of time and cutting out part of theacquired captured images when differences between the plurality ofcaptured images satisfy a preset condition.

According to the aspect of the invention, since the display image can begenerated and displayed based on the differences between the pluralityof captured images, motion images formed of a plurality of continuousimages can, for example, be used to generate display images based onchanges in the images and display the display images.

Another aspect of the invention is directed to a method for controllinga display apparatus worn on a body of a user before use and including adisplay section that transmits an outside scene and displays an image ina visually recognizable manner along with the outside scene, the methodincluding performing image capturing along directions of sight lines ofthe user, cutting and enlarging part of a captured image to generate adisplay image, and causing the display section to display the generateddisplay image.

According to the aspect of the invention, a display image based on acaptured image captured along the directions of the sight lines of theuser can be so displayed that the display image is superimposed on atarget object visible as the outside scene. A novel method for using animage displayed by the display apparatus can therefore be provided inthe form of an effective combination of an outside scene outside thedisplay apparatus and a displayed content, for example, by enhancing theway the user views the outside scene based on a captured image generatedby capturing the outside scene or presenting another way of viewing theoutside scene.

Still another aspect of the invention is directed to a programexecutable by a computer that controls a display apparatus worn on abody of a user before use and including a display section that transmitsan outside scene and displays an image in a visually recognizable manneralong with the outside scene, the program causing the computer tofunction as an image generation section that cuts and enlarges part of acaptured image generated by performing image capturing along directionsof sight lines of the user to generate a display image and an imagedisplay control section that causes the display section to display theimage generated by the image generation section.

According to the aspect of the invention, a display image based on acaptured image captured along the directions of the sight lines of theuser can be so displayed that the display image is superimposed on atarget object visible as the outside scene. A novel method for using animage displayed by the display apparatus can therefore be provided inthe form of an effective combination of an outside scene outside thedisplay apparatus and a displayed content, for example, by enhancing theway the user views the outside scene based on a captured image generatedby capturing the outside scene or presenting another way of viewing theoutside scene.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described with reference to the accompanyingdrawings, wherein like numbers reference like elements.

FIG. 1 is a descriptive diagram showing an exterior configuration of ahead mounted display according to a first embodiment.

FIG. 2 is a block diagram showing a functional configuration of the headmounted display.

FIG. 3 is a descriptive diagram showing an example of data stored in astorage section.

FIG. 4 is a flowchart showing action of the head mounted display.

FIGS. 5A to 5D are descriptive diagrams showing typical applicationexamples of the head mounted display. FIG. 5A shows an example of thefield of view of a user. FIGS. 5B and 5C shows examples of an image cutout from a captured image. FIG. 5D shows an example of the field of viewof the user in a case where a display image is displayed.

FIG. 6 is a flowchart showing action of the head mounted display.

FIGS. 7A to 7C show an example of the action of the head mounteddisplay. FIG. 7A shows an example of the environment in which the headmounted display is used, and FIGS. 7B and 7C show display examples.

FIG. 8 is a block diagram showing a functional configuration of a headmounted display according to a second embodiment.

FIGS. 9A and 9B show the configuration of key portions of an imagedisplay unit. FIG. 9A is a perspective view of the key portions of theimage display unit viewed from the side where user's head is present,and FIG. 9B describes the directions of sight lines.

FIG. 10 is a flowchart showing action of the head mounted display.

FIGS. 11A to 11D show display examples in the head mounted display.

FIG. 12 shows another display example in the head mounted display.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

First Embodiment

An embodiment to which the invention is applied will be described below.

FIG. 1 is a descriptive diagram showing an exterior configuration of ahead mounted display 100 according to a first embodiment. The headmounted display 100 is a display apparatus worn around a head and isalso called an HMD. The head mounted display 100 according to thepresent embodiment is an optically transmissive head mounted displaythat allows a user to not only visually recognize a virtual image butalso directly visually recognize an outside scene at the same time. Inthe present specification, a virtual image that the head mounted display100 allows the user to recognize is called a “display image” forconvenience. Further, outputting image light generated based on imagedata is also referred to as “displaying an image.”

The head mounted display 100 includes an image display unit 20, which isworn around user's head and allows the user to recognize a virtualimage, and a control unit 10, which controls the image display unit 20.The control unit 10 also functions as a controller that allows the userto operate the head mounted display 100. The image display unit 20 isalso simply called a “display section.”

The image display unit 20 is a wearable member worn around user's headand has a spectacle-like shape in the present embodiment. The imagedisplay unit 20 includes a right holder 21, a right display driver 22, aleft holder 23, a left display driver 24, a right optical image displaysection 26, a left optical image display section 28, a camera 61(imaging section), and a microphone 63. The right optical image displaysection 26 and the left optical image display section 28 are so disposedthat they are located in front of the right and left eyes of the userwho wears the image display unit 20. One end of the right optical imagedisplay section 26 and one end of the left optical image display section28 are connected to each other in a position corresponding to theportion between the eyes of the user who wears the image display unit20.

The right holder 21 is a member extending from an end ER of the rightoptical image display section 26, which is the other end thereof, to aposition corresponding to a right temporal region of the user who wearsthe image display unit 20. Similarly, the left holder 23 is a memberextending from an end EL of the left optical image display section 28,which is the other end thereof, to a position corresponding to a lefttemporal region of the user who wears the image display unit 20. Theright holder 21 and the left holder 23, which serve as temples (bows) ofspectacles, hold the image display unit 20 around user's head.

The right display driver 22 and the left display driver 24 are disposedon opposite sides of the head of the user who wears the image displayunit 20. In the following description, the right holder 21 and the leftholder 23 are also simply called “holders” in a collective manner. Theright display driver 22 and the left display driver 24 are also simplycalled “display drivers” in a collective manner. The right optical imagedisplay section 26 and the left optical image display section 28 arealso simply called “optical image display sections” in a collectivemanner.

The display drivers 22 and 24 include liquid crystal displays 241 and242 (hereinafter also referred to as “LCDs 241 and 242”), projectionsystems 251 and 252, and other components (see FIG. 2). Theconfiguration of the display drivers 22 and 24 will be described laterin detail. The optical image display sections 26 and 28, each of whichserves as an optical member, include light guide plates 261 and 262 (seeFIG. 2) and light control plates 20A. The light guide plates 261 and 262are made, for example, of a light transmissive resin and guide imagelight outputted from the display drivers 22 and 24 to user's eyes. Thelight control plates 20A are each a thin-plate-shaped optical elementand are so disposed that they cover the front side of the image displayunit 20 that faces away from the side where user's eyes are present.Each of the light control plates 20A can be a plate having lighttransmittance of substantially zero, a nearly transparent plate, a platethat transmits light but attenuates the amount of light, a plate thatattenuates or reflects light of a specific wavelength, or any of othervariety of optical component. Appropriate selection of opticalcharacteristics (such as light transmittance) of the light controlplates 20A allows adjustment of the amount of external light externallyincident on the right optical image display section 26 and the leftoptical image display section 28 and hence adjustment of visibility of avirtual image. In the present embodiment, a description will be made ofa case where the light control plates 20A are optically transmissiveenough to allow the user who wears the head mounted display 100 tovisually recognize an outside scene. The light control plates 20A alsoprotect the right light guide plate 261 and the left light guide plate262 and prevent the right light guide plate 261 and the left light guideplate 262 from being damaged, dirt from adhering thereto, and otherdefects from occurring.

The light control plates 20A may be configured to be attachable to anddetachable from the right optical image display section 26 and the leftoptical image display section 28, or a plurality of types of lightcontrol plates 20A may be exchangeably attachable. The light controlplates 20A may even be omitted.

The camera 61 is disposed at the end ER, which is the other end of theright optical image display section 26. The camera 61 captures an imageof an outside scene that is a scene of the outside, which is on the sideopposite user's eyes, to acquire an outside scene image. The camera 61in the present embodiment shown in FIG. 1 is a monocular camera and mayalternatively be a stereoscopic camera.

The image capturing direction of the camera 61, that is, the angle ofview thereof is defined on the front side of the head mounted display100, in other words, the direction in which the camera 61 captures animage of at least part of an outside scene in the direction of the fieldof view of the user who wears the head mounted display 100. Further, therange of the angle of view of the camera 61 can be set as appropriate,and it is preferable that the image capturing range of the camera 61covers the outside (outside scene) visually recognized by the userthrough the right optical image display section 26 and the left opticalimage display section 28. Further, it is more preferable that the imagecapturing range of the camera 61 is so set that the camera 61 cancapture an image of the entire field of view of the user through thelight control plates 20A.

The image display unit 20 further includes a connection section 40 forconnecting the image display unit 20 to the control unit 10. Theconnection section 40 includes a body cord 48, which is connected to thecontrol unit 10, a right cord 42, a left cord 44, and a connectionmember 46. The right cord 42 and the left cord 44 are two cords intowhich the body cord 48 bifurcates. The right cord 42 is inserted into anenclosure of the right holder 21 through a lengthwise end portion AP ofthe right holder 21 and connected to the right display driver 22.Similarly, the left cord 44 is inserted into an enclosure of the leftholder 23 through a lengthwise end portion AP of the left holder 23 andconnected to the left display driver 24.

The connection member 46 is disposed at the point where the body cord 48bifurcates into the right cord 42 and the left cord 44 and has a jackfor connecting an earphone plug 30. A right earphone 32 and a leftearphone 34 extend from the earphone plug 30. The microphone 63 isprovided in a position in the vicinity of the earphone plug 30. Anintegrated single code extends from the earphone plug 30 to themicrophone 63 and bifurcates at the microphone 63 into two codesconnected to the right earphone 32 and the left earphone 34,respectively.

Concrete specifications of the microphone 63 may be arbitrarilydetermined. The microphone 63 may be a directional microphone or anomni-directional microphone. Examples of the directional microphone mayinclude a unidirectional (cardioid) microphone, a narrow directivity(super cardioid) microphone, a sharp directivity (hypercardioid)microphone, and an ultra-directivity (ultra cardioid) microphone. Themicrophone 63, when it has directivity, may be so configured that itcollects and detects voice coming along the directions of the sightlines of the user who wears the head mounted display 100 in aparticularly sensitive manner. In this case, to ensure the directivityof the microphone 63, the microphone 63 or a part that accommodates themicrophone 63 may have a structural feature. For example, in the exampleshown in FIG. 1, the microphone 63 and the connection member 46 may beso designed that a sound collection portion of the microphone 63 facesthe directions of the sight lines of the user who wears the rightearphone 32 and the left earphone 34. The microphone 63 may instead beembedded in the right holder 21 or the left holder 23. In this case,opening a sound-collecting hole in a front-side surface of the rightholder 21 or the left holder 23, that is, a surface aligned with theright optical image display section 26 or the left optical image displaysection 28 allows directivity corresponding to the directions of user'ssight lines to be ensured. The directions of user's sight lines are, inother words, for example, the directions that the right optical imagedisplay section 26 and the left optical image display section 28 face,the direction along which the user faces the center of the field of viewthat the user views through the right optical image display section 26and the left optical image display section 28, and the image capturingdirection of the camera 61. Further, the direction of the directivity ofthe microphone 63 may be configured to be variable. In this case, thedirections of user's sight line may be detected, and the direction ofthe directivity of the microphone 63 may be adjusted so as to coincidewith the directions of the sight lines.

The right cord 42 and the left cord 44 can instead be integrated witheach other into a single cord. Specifically, the wire in the right cord42 may be caused to pass through the internal space in the body of theimage display unit 20, brought into the left holder 23, and joined withthe wire in the left cord 44, and the two wires are coated with a resinto form a single cord.

The image display unit 20 and the control unit 10 transmit a variety ofsignals to each other via the connection section 40. Connectors (notshown) that engage with each other are provided at the end of the bodycord 48 that is opposite the end where the connection member 46 ispresent and at an end of the control unit 10. Causing the connectors atthe body cord 48 and the control unit 10 to engage with each other anddisengage from each other allows the control unit 10 and the imagedisplay unit 20 to be connected to each other and disconnected from eachother. Each of the right cord 42, the left cord 44, and the body cord 48can, for example, be a metal cable or an optical fiber.

The control unit 10 is an apparatus for controlling the head mounteddisplay 100. The control unit 10 has a group of switches including afinalizing key 11, a lighting portion 12, a display switch key 13, aluminance switch key 15, a direction key 16, a menu key 17, and a powerswitch 18. The control unit 10 further includes a track pad 14, on whichthe user performs touch operation with a finger.

The finalizing key 11 detects user's pressing operation and outputs asignal that finalizes action corresponding to the operation performed onthe control unit 10. The lighting portion 12 notifies the user of thestate of action of the head mounted display 100 in the form of thelighting state. An example of the state of action of the head mounteddisplay 100 is whether it is powered on or off. The lighting portion 12is formed, for example, of an LED (light emitting diode). The displayswitch key 13 detects user's pressing operation and outputs a signalthat switches, for example, content motion image display mode between a3D mode and a 2D mode.

The track pad 14 detects operation performed by user's finger on anoperation surface of the track pad 14 and outputs a signal according toa result of the detection. Examples of the track pad 14 may include avariety of types of track pad, such as an electrostatic type, a pressuredetection type, and an optical type. The luminance switch key 15 detectsuser's pressing operation and outputs a signal that increases ordecreases the luminance of an image displayed by the image display unit20. The direction key 16 detects user's pressing operation performed ona portion corresponding to any of the upward, downward, rightward, andleftward directions and outputs a signal according to a result of thedetection. The power switch 18 detects user's operation of sliding theswitch and powers on or off the head mounted display 100.

FIG. 2 is a functional block diagram of portions that form a displaysystem 1 according to the embodiment.

The display system 1 includes an external apparatus OA and the headmounted display 100, as shown in FIG. 2. The external apparatus OA is,for example, a personal computer (PC), a mobile phone terminal, and agame console. The external apparatus OA is used as an image supplyapparatus that supplies the head mounted display 100 with images.

The control unit 10 of the head mounted display 100 includes a controlsection 140, an operation section 135, an input information acquisitionsection 110, a storage section 120, a power supply 130, an interface180, a transmitter (Tx) 51, and a transmitter (Tx) 52.

The operation section 135 detects user's operation. The operationsection 135 includes the keys and switches shown in FIG. 1, finalizingkey 11, the display switch key 13, the track pad 14, the luminanceswitch key 15, the direction key 16, the menu key 17, and the powerswitch 18.

The input information acquisition section 110 acquires a signalaccording to a user's operation input. Examples of the signal accordingto the operation input include inputs corresponding to operationperformed on the track pad 14, the direction key 16, and the powerswitch 18.

The power supply 130 supplies the portions that form the head mounteddisplay 100 with electric power. The power supply 130 can, for example,be a secondary battery.

The storage section 120 stores a variety of computer programs. Thestorage section 120 is formed, for example, of a ROM and a RAM. Thestorage section 120 may store image data to be displayed in the imagedisplay unit 20 of the head mounted display 100.

The storage section 120 stores detection feature data 124, which atarget detection section 171, which will be described later, refers to,and display setting data 126, which is processed by an image displaycontrol section 176, which will be described later.

The interface 180 connects the variety of external apparatus OA, whichare content supply sources, to the control unit 10. The interface 180can be an interface that supports wired connection, such as a USBinterface, a micro-USB interface, and a memory card interface.

The control section 140, which reads and executes the computer programsstored in the storage section 120, achieves a function of each portion.That is, the control section 140 functions as an operating system (OS)150, an image processing section 160, a voice processing section 170, atarget detection section 171, a distance detection section 173, an imagedisplay control section 176, an image generation section 177, and adisplay control section 190.

A three-axis sensor 113, a GPS 115, and a communication section 117 areconnected to the control section 140. The three-axis sensor 113 is athree-axis acceleration sensor, and the control section 140 can acquirevalues detected with the three-axis sensor 113. The GPS 115 includes anantenna (not shown), receives GPS (global positioning system) signals,and determines the current position of the control unit 10. The GPS 115outputs the current position and current time determined based on theGPS signals to the control section 140. The GPS 115 may further have afunction of acquiring the current time based on information contained inthe GPS signals and correcting the time clocked by the control section140 in the control unit 10.

The communication section 117 performs wireless data communication thatcomplies with wireless LAN (WiFi (registered trademark)), Miracast(registered trademark), Bluetooth (registered trademark), or any otherstandard.

When the external apparatus OA is wirelessly connected to thecommunication section 117, the control section 140 acquires content datavia the communication section 117 and controls and causes the imagedisplay unit 20 to display an image. On the other hand, when theexternal apparatus OA is wired to the interface 180, the control section140 acquires content data through the interface 180 and control andcauses the image display unit 20 to display an image. The communicationsection 117 and the interface 180 are therefore hereinafter collectivelyreferred to as data acquisition sections DA.

The data acquisition sections DA acquire content data to be displayed bythe head mounted display 100 from the external apparatus OA. The contentdata contains image data that will be described later.

The image processing section 160 acquires an image signal contained inthe content. The image processing section 160 separates a vertical syncsignal VSync, a horizontal sync signal HSync, and other sync signalsfrom the acquired image signal. Further, the image processing section160 produces a clock signal PCLK, for example, by using a PLL (phaselocked loop) circuit (not shown) in accordance with the cycles of theseparated vertical sync signal VSync and horizontal sync signal HSync.The image processing section 160 converts the analog image signal fromwhich the sync signals are separated into a digital image signal, forexample, by using an A/D conversion circuit (not shown). The imageprocessing section 160 then stores the converted digital image signal asimage data on an image to be displayed (Data in FIG. 2) in a DRAM in thestorage section 120 on a frame basis. The image data is, for example,RGB data.

The image processing section 160 may perform as required resolutionconversion, luminance adjustment, chroma adjustment, and a variety ofother types of color tone correction, keystone correction, and othertypes of image processing on the image data.

The image processing section 160 transmits the produced clock signalPCLK, the vertical sync signal VSync, the horizontal sync signal HSync,and the image data Data stored in the DRAM in the storage section 120via the transmitters 51 and 52. The image data Data transmitted via thetransmitter 51 is also called “image data for the right eye,” and theimage data Data transmitted via the transmitter 52 is also called “imagedata for the left eye.” Each of the transmitters 51 and 52 functions asa transceiver for serial transmission between the control unit 10 andthe image display unit 20.

The display control section 190 produces control signals that controlthe right display driver 22 and the left display driver 24.Specifically, the display control section 190 separately controls, withthe aid of the control signals, for example, whether or not a right LCDcontrol section 211 is instructed to drive the right LCD 241, whether ornot a right backlight control section 201 is instructed to drive a rightbacklight 221, whether or not a left LCD control section 212 isinstructed to drive the left LCD 242, and whether or not a leftbacklight control section 202 is instructed to drive a left backlight222. The display control section 190 thus controls and causes each ofthe right display driver 22 and the left display driver 24 to generateand output image light. For example, the display control section 190causes both the right display driver 22 and the left display driver 24to generate image light, only one of them to generate image light, orneither of them to generate image light.

The display control section 190 transmits the control signals to theright LCD control section 211 and the left LCD control section 212 viathe transmitters 51 and 52. The display control section 190 furthertransmits the control signals to the right backlight control section 201and the left backlight control section 202.

The image display unit 20 includes the right display driver 22, the leftdisplay driver 24, the right light guide plate 261 as the right opticalimage display section 26, the left light guide plate 262 as the leftoptical image display section 28, the camera 61, a vibration sensor 65,and a nine-axis sensor 66.

The vibration sensor 65 is formed of an acceleration sensor and disposedin the image display unit 20, as shown in FIG. 1. In the example shownin FIG. 1, the vibration sensor 65 is built in a portion in the rightholder 21 and in the vicinity of the end ER of the right optical imagedisplay section 26. The vibration sensor 65 detects vibration producedwhen the user knocks the end ER (performs knock operation) and outputs aresult of the detection to the control section 140. Based on a result ofthe detection performed by the vibration sensor 65, the control section140 detects the knock operation performed by the user.

The nine-axis sensor 66 is a motion sensor that detects acceleration(three axes), angular velocity (three axes), and terrestrial magnetism(three axes). The nine-axis sensor 66, which is provided in the imagedisplay unit 20, detects motion of the head of the user who wears theimage display unit 20 around the head. Since the orientation of theimage display unit 20 is identified based on the detected motion ofuser's head, the control section 140 can estimate the directions ofuser's sight lines.

The right display driver 22 includes a receiver (Rx) 53, the rightbacklight (BL) controlling section 201 and the right backlight (BL) 221,which function as a light source, the right LCD control section 211 andthe right LCD 241, which function as a display device, and the rightprojection system 251. The right backlight control section 201 and theright backlight 221 function as a light source. The right LCD controlsection 211 and the right LCD 241 function as a display device. Theright backlight control section 201, the right LCD control section 211,the right backlight 221, and the right LCD 241 are also collectivelyreferred to as an “image light generation unit.”

The receiver 53 functions as a receiver for serial transmission betweenthe control unit 10 and the image display unit 20. The right backlightcontrol section 201 drives the right backlight 221 based on the inputtedcontrol signal. The right backlight 221 is, for example, an LED, anelectroluminescence (EL) device, or any other light emitter. The rightLCD control section 211 drives the right LCD 241 based on the clocksignal PCLK, the vertical sync signal VSync, the horizontal sync signalHSync, and the image data for the right eye Data1, which are inputtedvia the receiver 53. The right LCD 241 is a transmissive liquid crystalpanel having a plurality of pixels arranged in a matrix.

The right projection system 251 is formed of a collimator lens thatconverts the image light outputted from the right LCD 241 into aparallelized light flux. The right light guide plate 261 as the rightoptical image display section 26 reflects the image light having exitedout of the right projection system 251 along a predetermined opticalpath and eventually guides the image light to user's right eye RE. Theright projection system 251 and the right light guide plate 261 are alsocollectively referred to as a “light guide unit.”

The left display driver 24 has the same configuration as that of theright display driver 22. The left display driver 24 includes a receiver(Rx) 54, the left backlight (BL) control section 202 and the leftbacklight (BL) 222, which function as a light source, the left LCDcontrol section 212 and the left LCD 242, which function as a displaydevice, and the left projection system 252. The left backlight controlsection 202 and the left backlight 222 function as a light source. Theleft LCD control section 212 and the left LCD 242 function as a displaydevice. The left backlight control section 202, the left LCD controlsection 212, the left backlight 222, and the left LCD 242 are alsocollectively referred to as an “image light generation unit.” The leftprojection system 252 is formed of a collimator lens that converts theimage light outputted from the left LCD 242 into a parallelized lightflux. The left light guide plate 262 as the left optical image displaysection 28 reflects the image light having exited out of the leftprojection system 252 along a predetermined optical path and eventuallyguides the image light to user's left eye LE. The left projection system252 and the left light guide plate 262 are also collectively referred toas a “light guide unit.”

When the user views an outside scene through the right optical imagedisplay section 26 and the left optical image display section 28, thehead mounted display 100 displays an image based on image data in such away that the image is superimposed on the outside scene.

The target detection section 171 controls and causes the camera 61 toperform image capturing to acquire a captured image. The captured imageis outputted as color image data or monochrome image data from thecamera 61. Instead, the camera 61 may output an image signal, and thetarget detection section 171 may generate image data that complies witha predetermined file format from the image signal.

The target detection section 171 analyzes the acquired captured imagedata to detect a target object contained in the captured image data. Thetarget object is an object or a person present in the image capturingdirection of the camera 61, that is, in the directions of the user'ssight lines.

The target detection section 171 searches the captured image for animage that conforms to the detection feature data 124 and detects aconforming image as an image of the target object.

FIG. 3 shows an example of the configuration of the detection featuredata 124 and the display setting data 126 stored in the storage section120.

The detection feature data 124 is data on the amount of feature of animage detected from the captured image. In the present embodiment, thedetection feature data 124 contains single image feature data 124 a andmotion image feature data 124 b. The single image feature data 124 a isdata on the amount of feature applied to a single captured image. Thetarget detection section 171, when it uses the single image feature data124 a, performs a matching process of detecting a portion that conformsto the single image feature data 124 a in a single captured image.

The motion image feature data 124 b is data on the amount of featureapplied to a plurality of captured images. The target detection section171, when it uses the motion image feature data 124 b, acquires capturedimages from the camera 61 multiple times and determines differencesbetween the plurality of acquired captured images. The camera 61 canperform image capturing at predetermined time intervals and outputcaptured image data whenever the image capturing is performed. That is,the camera 61 functions as a motion image camera that performs imagecapturing at a frame rate corresponding to the time intervals describedabove. In this case, the target detection section 171 can determinechanges in the motion images by determining differences between theplurality of captured images captured with the camera 61 at differentpoints of time. It can be said that the motion image feature data 124 bis data on the amount of feature applied to changes in motion images.The target detection section 171 performs a matching process ofdetecting a portion that conforms to the motion image feature data 124 bin the differences between the captured images.

In the present embodiment, whether the detection feature data 124 usedby the target detection section 171 is the single image feature data 124a or the motion image feature data 124 b is set in advance. The targetdetection section 171, when it is set to use the single image featuredata 124 a, acquires a single captured image from the camera 61 andperforms the matching process. In contrast, the target detection section171, when it is set to use the motion image feature data 124 b,temporarily stores, whenever acquiring a captured image from the camera61, the captured image in the storage section 120. The target detectionsection 171 then acquires the plurality of captured images stored in thestorage section 120, determines differences between the acquired images,and performs the matching process based on the motion image feature data124 b on the differences.

Based on a result of the matching process, the target detection section171 identifies a target object contained in the captured image capturedwith the camera 61. That is, the target detection section 171, when itdetects an area that conforms to the single image feature data 124 a inthe captured image or an area that conforms to the motion image featuredata 124 b in the differences between the captured images, sets thesubject contained in the detected area to be a target object.

The distance detection section 173 determines the distance to the targetobject detected by the target detection section 171. For example, thedistance detection section 173 determines the distance to the targetobject based on the size of the image of the target object detected bythe target detection section 171 in the captured image from the camera61.

The head mounted display 100 may instead include a distance meter thatdetects the distance to a target object by using a laser beam or anultrasonic wave. The distance meter, for example, includes a lightsource that emits a laser beam and a light receiver that receives thelaser beam emitted from the light source and reflected off a targetobject, and the distance meter detects the distance to the target objectbased on the state of the received laser beam. The distance meter mayinstead be, for example, a distance meter based on an ultrasonic wave.That is, the distance meter may include a sound source that emits anultrasonic wave and a detector that detects the ultrasonic wavereflected off a target object, and the distance meter may detect thedistance to the target object based on the reflected ultrasonic wave.The distance meter can still instead be a combination of the distancemeter using a laser beam and the distance meter using an ultrasonicwave. A distance meter of this type is preferably provided in the rightholder 21 or the right display driver 22 in the image display unit 20and may be disposed in a surface aligned with the light control plates20A with the distance meter facing forward. The direction along whichthe distance meter measures the distance desirably coincides with thedirection of user's sight lines, as in the case of the image capturingdirection of the camera 61.

The distance detection section 173 detects the distance from the camera61 or the distance meter to a target object, and the distance can beconsidered to be the distance from the user of the head mounted display100 to the target object.

Further, in a case where a target object detected by the targetdetection section 171 is not a specific object, the center of a capturedimage, for example, is assumed to be an imaginary target object, and thedistance detection section 173 detects the distance to the imaginarytarget object.

The image generation section 177 extracts and processes part of acaptured image from the camera 61 to generate a display image. In thepresent embodiment, the image generation section 177 cuts out apredetermined-size area containing a feature detected by the targetdetection section 171 from the captured image to generate a displayimage. When the target detection section 171 uses the motion imagefeature data 124 b to detect a target object from a plurality ofcaptured images, the image generation section 177 cuts out an areacontaining the target object detected by the target detection section171 from the latest captured image.

The display image generated by the image generation section 177 isdisplayed in displayable areas of the image display unit 20 under thecontrol of the image display control section 176. The size according towhich the display image is displayed is set in advance. For example, thesize is roughly so set that the display image does not prevent the userto view an outside scene. The image generation section 177 thereforecuts out part of a captured image in accordance with the display size ofthe display image.

The image generation section 177 generates a display image in such a waythat part of a captured image containing a feature detected by thetarget detection section 171 is displayed in a readily viewable manner.The process carried out by the image generation section 177 is thereforenot limited to the process of enlarging the area extracted from thecaptured image as described above. For example, the display image may beformed by surrounding the target object, which is present in the areaextracted from the captured image, with a circular or rectangular frameor a frame having any other shape. Still instead, for example, thedisplay image may be generated by adding a balloon, an arrow, or anyother decoration to the target object so that the user readilyrecognizes the target object. The decoration in this case is an imagestored in advance for the decoration purpose and can be called anauxiliary image or an enhancement image. Further, the luminance or colortone of the area containing the target object may be changed, or theluminance of the target object and a portion therearound may, forexample, be increased.

To add a decoration to a target object detected based on the singleimage feature data 124 a or the motion image feature data 124 b, data onthe decoration may be contained in the single image feature data 124 aand the motion image feature data 124 b.

The image generation section 177, when it cuts out an image containing atarget object detected by the target detection section 171, may cut outan image of the target object and a portion therearound. In this case,since the user can clearly see that the display image is an enlargedimage of part of the captured image, the user can view, for example, afar portion that is hard to see in the form of an enlarged image.Further, the image generation section 177 may cut out part of a capturedimage along the contour of a target object. In this case, the cutoutimage can be combined with another background image and the combinedimage can be displayed, or the cutout image can be processed so as toform 3D (stereoscopic) image and the 3D (stereoscopic) image can bedisplayed in the image display unit 20.

The image generation section 177 further determines an enlargementfactor based on the distance detected by the distance detection section173. When an image of the portion cut out from a captured image isdisplayed under the control of the image display control section 176,the image generation section 177 processes the cutout image so as toallow the user to view an enlarged version of the cutout image. Theimage generation section 177 therefore determines the size of the imagecut out from the captured image based on the size (resolution or numberof image dots) in accordance with which the display image is displayedin the displayable areas of the image display unit 20 and theenlargement factor. The image generation section 177 then carries out anenlargement process of converting the size of the cutout image inaccordance with the display size (or resolution conversion process). Theimage cut out from the captured image is thus enlarged, whereby the usercan view an enlarged version of the cutout image.

The size of the display image enlarged by the image generation section177 and visually recognized by the user is affected by thespecifications of the image display unit 20. To allow the imagegeneration section 177 appropriately to manage and adjust theenlargement factor in accordance with which an image cut out from acaptured image, it is, for example, preferable to set in advance theimage size in a case where the size of an image cut out from a capturedimage is equal to the size of an outside scene in the field of view ofthe user (that is, enlargement factor is one). That is, the size of animage cut out by the image generation section 177 in a case where thesize of a display image visually recognized by the user is not enlargedor reduced is set in advance. With reference to the size, the imagegeneration section 177 cuts out an image in accordance with a targetenlargement factor and enlarges the image.

The enlargement factor in accordance with which the image generationsection 177 enlarges an image may, for example, be determined incorrespondence with the distance detected by the distance detectionsection 173.

FIG. 3 shows an example in which the display setting data 126 stored inthe storage section 120 contains enlargement factor setting data 126 a,which relates the distance to a target object detected by the distancedetection section 173 to the enlargement factor. The enlargement factorsetting data 126 a is data that specifies the distance detected by thedistance detection section 173 and the enlargement factor and contains,for example, a table that allows stepwise determination of theenlargement factor or a computing equation and parameters that allowcalculation of the enlargement factor from the distance. Based on theenlargement factor setting data 126 a, the image generation section 177can determine the enlargement factor from the distance detected by thedistance detection section 173. The enlargement factor setting data 126a may further contain a setting used in a case where the imagegeneration section 177 does not perform the enlargement or a settingused in a case where the image generation section 177 generates nodisplay image. For example, the enlargement factor setting data 126 amay contain a setting that does not allow generation or display of adisplay image when the distance to a target object detected by thedistance detection section 173 is shorter than a preset distance. Whenthe image generation section 177 does not generate or output data on adisplay image, the image display control section 176 displays no displayimage.

The display setting data 126 may further contain data set in advance inrelation to image display in addition to the enlargement factor settingdata 126 a, such as setting value data on a default display size of animage displayed in the image display unit 20 and setting value data ondisplay color adjustment.

The image display control section 176 controls the display controlsection 190 to cause the image display unit 20 to display an image basedon data on a display image generated by the image generation section177.

The voice processing section 170 acquires a voice signal contained inthe content, amplifies the acquired voice signal, and supplies theamplified voice signal to a speaker (not shown) in the right earphone 32and a speaker (not shown) in the left earphone 34, which are connectedto the connection member 46. For example, when a Dolby (registeredtrademark) system is employed, relevant processing is performed on thevoice signal, and the right earphone 32 and the left earphone 34 outputsounds differentiated in terms, for example, of frequency.

The voice processing section 170 further acquires voice collected by themicrophone 63, converts the collected voice into digital voice data, andcarries out a process relating to voice. For example, the voiceprocessing section 170 may perform speaker recognition in which featuresare extracted from the acquired voice and modeled to individuallyrecognize voices of a plurality of persons and identify a speaker on avoice basis.

The three-axis sensor 113, the GPS 115, and the communication section117 are connected to the control section 140. The three-axis sensor 113is a three-axis acceleration sensor, and the control section 140 canacquire values detected with the three-axis sensor 113 to detect motionof the control unit 10 and the direction of the motion.

The GPS 115 includes an antenna (not shown), receives GPS (globalpositioning system) signals, and determines the current position of thecontrol unit 10. The GPS 115 outputs the current position and currenttime determined based on the GPS signals to the control section 140. TheGPS 115 may further have a function of acquiring the current time basedon information contained in the GPS signals and correcting the timeclocked by the control section 140 in the control unit 10.

The communication section 117 performs wireless data communication thatcomplies with a wireless LAN (WiFi (registered trademark)) standard or aBluetooth (registered trademark) standard.

The interface 180 connects a variety of image supply apparatus OA, whichare content supply sources, to the control unit 10. A content suppliedby the image supply apparatus OA contains still images or motion imagesand may contain voice. Examples of the image supply apparatus OA mayinclude a personal computer (PC), a mobile phone terminal, and a gameconsole. The interface 180 can, for example, be a USB interface, amicro-USB interface, and a memory card interface.

The image supply apparatus OA can instead be connected to the controlunit 10 via a wireless communication line. In this case, the imagesupply apparatus OA performs wireless communication with thecommunication section 117 and transmits data in a content by using awireless communication technology, such as Miracast (registeredtrademark).

FIG. 4 is a flowchart showing action of the head mounted display 100. Inthe action, when the user views an outside scene through the rightoptical image display section 26 and the left optical image displaysection 28, the image display unit 20 displays part of a captured imagegenerated by capturing the outside scene.

FIGS. 5A to 5D are descriptive diagrams showing how the head mounteddisplay 100, which acts as shown in FIG. 4, displays an image. FIG. 5Ashows an example of a field of view VR of the user. FIGS. 5B and 5C showexamples of an image cut out from a captured image. FIG. 5D shows anexample of the field of view VR of the user in a case where a displayimage is displayed.

In the control section 140 of the head mounted display 100, the targetdetection section 171 causes the camera 61 to perform image capturing(step S11) to acquire a captured image (step S12). The target detectionsection 171 acquires the single image feature data 124 a or the motionimage feature data 124 b from the storage section 120 (step S13) andevaluates whether a target in the acquired detection feature data is asingle captured image or a plurality of captured images (step S14). Theevaluation can be performed based on whether the single image featuredata 124 a has been specified or the motion image feature data 124 b hasbeen specified as the detection feature data to be used.

When a single captured image is processed (step S14; YES), the targetdetection section 171 performs the matching process on the capturedimage acquired in step S12 by using the single image feature data 124 a(step S15) and proceeds to step S18.

When a plurality of captured images are processed (step S14; NO), thetarget detection section 171 determines differences between the capturedimages acquired in step S12 and captured images captured in the past andtemporarily stored in the storage section 120 (step S16). The targetdetection section 171 performs the matching process based on the motionimage feature data 124 b on the determined differences (step S17) andproceeds to step S18.

In step S18, the target detection section 171 determines an area to becut out from the captured image. The area to be determined in thisprocess is an area containing the target object detected by the targetdetection section 171, but a detailed cutout range is determined by theimage generation section 177 in step S21, which will be described later.The area determined in step S18 is therefore a temporal area and may,for example, be the position of the center of the area to be cut out orthe contour of the target object.

The example shown in FIG. 5A is a case where the user views a footballgame from a spectator seat. In the example, a football field F and aplurality of players FP who play in the field F are visible, and a scenein which one player FP is keeping a ball B is visible. In the example,since the user is present in a position far away from the field F, theplayers FP and the ball B in the field of view VR are seen as smallobjects.

The image capturing range (angle of view) of the camera 61 is, forexample, equal to the field of view VR, and a captured image thereforecontains the field F, the players FP, and the ball B.

It is now assumed that the target detection section 171 performs thematching process on the captured image based on the single image featuredata 124 a to detect an image of the ball B as a target object. In thiscase, the target detection section 171 detects the ball B in step S15,and an area containing the ball B is determined as the cutout area instep S18.

The distance detection section 173 subsequently detects the distance tothe target object contained in the area determined by the targetdetection section 171 in step S18 (step S19).

The image generation section 177 then determines the enlargement factorbased on the distance detected by the distance detection section 173 andthe enlargement factor setting data 126 a stored in the storage section120 (step S20). The image generation section 177 determines the size andposition of an image to be cut out from the captured image based on thedetermined enlargement factor, actually cuts out an image, and outputsthe cutout image as a display image (step S21). In step S21, the imagegeneration section 177 may perform the size conversion, resolutionconversion, or other types of processing described above. Further, theimage generation section 177 outputs no display image when theenlargement factor setting data 126 a corresponding to the distancedetected by the distance detection section 173 is so set that no displayis performed.

FIGS. 5B and 5C show examples of the display image outputted from theimage generation section 177. The image generation section 177 cut outsan area containing an image of the ball B from the captured image inaccordance with settings set in advance in relation to the imagecutting-out operation. The settings set in advance in relation to theimage cutting-out operation include the aspect ratio of an image to becut out and the size of the image in a case where the enlargement factoris one. In the example shown in FIG. 5B, an area containing the ball Band the whole body of the player FP who is keeping the ball B is cutout, and the cutout image is outputted as the display image. In theexample shown in FIG. 5C, an area containing the ball B and a portioncontaining the ball B and having a predetermined size is cut out, andthe cutout image is outputted as the display image. In the example shownin FIG. 5C, only part of the body of the player FP who is keeping theball B is contained. When the single image feature data 124 a is used asdata on the amount of feature for detecting the ball B, whether an imageto be cut out by the image generation section 177 contains the wholebody of the player FP or part of the body of the player FP is determinedbased on the enlargement factor and other factors.

The image display control section 176 stores the display image outputtedfrom the image generation section 177 in the storage section 120 (stepS22). The storage section 120 stores display images in a time sequentialmanner. Therefore, when the processes shown in step S11 to S24 in FIG. 4are repeatedly carried out, the storage section 120 stores a pluralityof display images in a time sequential manner. Motion images can bereproduced through continuous display of the plurality of displayimages. For example, the control section 140 can reproduce and displaythe display images stored in the storage section 120 in accordance withinstruction operation performed on the operation section 135.

The image display control section 176 outputs the display imageoutputted from the image generation section 177 in step S21 to thedisplay control section 190, which causes the image display unit 20 todisplay the display image (step S23).

The control section 140 then evaluates whether or not the displayoperation is terminated (step S24). When the display operation iscontinued (step S24; NO), the control section 140 returns to step S11.When the display operation is terminated in accordance, for example,with operation detected by the operation section 135 (step S24; YES),the control section 140 causes the display control section 190 to stopthe display operation and terminates the entire process.

FIG. 5D shows a state in which the image display unit 20 displays thedisplay image. The field F is visible as an outside scene in the fieldof view VR, as in FIG. 5A, and an image IV is so displayed that it issuperimposed on part of the field of view VR. The image IV is thedisplay image displayed by the image display unit 20. Since the image IVis an image generated by enlarging part of the outside scene, the usercan view the outside scene viewed through the image display unit 20 andpart of the outside scene at the same time.

The target detection section 171, when it repeatedly carries out theprocesses in step S11 to S17 in FIG. 4, may track the target object thatthe target detection section 171 has once detected. That is, the targetdetection section 171 detects a target object based on the single imagefeature data 124 a or the motion image feature data 124 b, then monitorschanges in captured images from the camera 61, and tracks the motion ofthe detected target object. The target detection section 171 can thusdetect the target object from the captured images under the detectionoperation without performing the matching processes in steps S15 andS17. The approach described above is effective when the camera 61captures motion images at a predetermined frame rate (30 frames/second,for example) and the target detection section 171 detects a targetobject on a frame basis, because the processing burden can be reduced.

Each of the single image feature data 124 a and the motion image featuredata 124 b is not limited to the amount of feature for detecting aspecific target object. For example, the single image feature data 124 acan be the amount of feature for detecting a captured image containing aspecific feature, and the motion image feature data 124 b can be theamount of feature for detecting motion images that change in a specificmanner. In this case, the target detection section 171, when it detectsa feature that conforms to the single image feature data 124 a or themotion image feature data 124 b in a captured image, sets a portioncontaining the feature to be a target object. In this case, the targetobject is not a specific object but is part of the captured image. Theimage generation section 177 may cut out the portion of the capturedimage that has been detected as the target object by the targetdetection section 171 and set the portion to be a display image.

Instead, the single image feature data 124 a and the motion imagefeature data 124 b may contain data on the amount of feature for aspecific target object that allows detection of a specific one targetobject or a small number of target objects or may contain data on theamount of feature for general purposes that allows detection of aplurality of types of target object similar to each other.

In the examples shown in FIGS. 5B and 5C, for example, the targetdetection section 171 detects the ball B based on the single imagefeature data 124 a. The single image feature data 124 a may be data onthe amount of feature for a specific target object that allows detectionof a ball having a specific pattern (including color and shape ofpattern) or may be data on the amount of feature for general purposesthat allows detection of a spherical object. When the single imagefeature data 124 a is data on the amount of feature for generalpurposes, the data may, for example, be the amount of feature thatidentifies only the shape, the amount of feature that identifies theshape and color, or the amount of feature that identifies one of aplurality of colors. Further, the single image feature data 124 a maycontain data that specifies the range of the difference from anacceptable amount of feature in addition to the data on the amount offeature. The same holds true for the motion image feature data 124 b.

Specifically, when the detection feature data 124 is data on the amountof feature for general purposes that accepts variations in the face,hair, and clothing (such as uniform, shoes, cap, and bibs) of a specificperson, a plurality of target objects can be detected, and further, thestate of sunshine or illumination, brightness in the environment, and achange in color in a captured image due to sunlight can be handled.Further, having detected a target object, the target detection section171 may track the target object in captured image data that are keptcaptured. In this case, when an image of the target object in thecapture image data does not conform to the amount of feature during thetracking of the target object, the amount of feature may be calibrated.As a result, both in the case where data on the amount of feature for aspecific target is used and the case where data on the amount of featurefor general purposes is used, the target object can be detected and keptdetected even when the color or the shape of the target object in acaptured image deviates from the amount of feature due to sunshine andother factors.

FIG. 6 is a flowchart showing action of the head mounted display 100 andshows action of adjusting display operation in correspondence with theposition of the user who wears the head mounted display 100. FIGS. 7A to7C are descriptive diagrams of the same action as that in shown in FIG.6. FIG. 7A shows an example of the environment where the head mounteddisplay is used, and FIGS. 7B and 7C show display examples.

The action shown in FIG. 6 is a process of adjusting the position wherethe image display unit 20 displays the image IV and the size of theimage IV in correspondence with the position of the user. As describedabove, the head mounted display 100 is an apparatus that allows the userto view a captured image from the camera 61 and displayed in the imagedisplay unit 20 while viewing an outside scene. The direction in whichthe user views an outside scene through the image display unit 20 isstrongly affected by the relative positional relationship between theuser and a target object viewed by the user.

For example, consider a case shown in FIG. 7A where the user wears anduses the head mounted display 100 in a stadium ST having spectator seatsS arranged in multiple floor layers (four layers in FIG. 7A). When theuser is present in the fourth or third floor or any other upper-layerfloor, the user looks down to view the field F. The direction alongwhich the user desires to look is present below display areas of theright light guide plate 261 and the left light guide plate 262. In thiscase, the image display control section 176 displays a display image,for example, in a position above the display areas separate from thedirection in which the user desires to look, so that the display imagedoes not bother the user. The example shown in FIG. 7B is a case wherethe user views the players FP in the field F from an upper-layer floor,and the image display control section 176 displays the image IV in aportion above the display areas. In this case, the user can view theplayers FP in the field of view VR in a state in which the image IV doesnot bother the user. The example shown in FIG. 7C is a case where theuser views the players FP in the field F from a lower-layer floor, andthe image display control section 176 displays the image IV in a portionbelow the display areas. In this case, the user's sight lines approachthe frontward direction, the image display control section 176 displaysa reduced image IV in a position separate from the frontward direction.The user can therefore view the players FP in the field of view VR in astate in which the image IV does not bother the user.

The action shown in FIG. 6 can be performed at an arbitrary point oftime after the display operation starts in step S23 in FIG. 4 and when apreset condition is satisfied or in response to user's operationperformed on the control unit 10.

The image display control section 176 determines the relative positionalrelationship between a target object contained in an outside sceneviewed by the user and the head mounted display 100 (step S31). Theposition of the head mounted display 100 may be determined based, forexample, on the position detected by the GPS 115. The position of thehead mounted display 100 may instead be determined based on the distanceto the target object detected by the distance detection section 173 fromcaptured image data from the camera 61. Still instead, in a state inwhich a position identification system using WiFi (registered trademark)is in operation in the environment in which the head mounted display 100is used, the communication section 117 may identify the position of thehead mounted display 100 by performing WiFi communication. Stillinstead, when the position of the head mounted display 100 is specifiedby input operation performed on the operation section 135, the specifiedposition may be acquired as the position of the head mounted display100.

The image display control section 176 refers to the display setting data126 stored in the storage section 120 and acquires settings relating todisplay operation corresponding to the position determined in step S31(step S32). The settings include the position where the image IV isdisplayed, a defined value or a maximum value of the display size of theimage IV, and other parameters. The image display control section 176adjusts the display size of the image IV and the position where theimage IV is displayed in accordance with the settings acquired in stepS32 (step S33) and evaluates whether or not the display operation isterminated (step S34).

The control section 140 repeatedly carries out the processes in stepsS31 to S33 as long as the display operation is not terminated (step S34;NO) and terminates the entire process when the display operation isterminated (step S34; YES).

As described above, the head mounted display 100 according to the firstembodiment to which the invention is applied is a display apparatus wornon use's body before use and includes the image display unit 20, whichtransmits an outside scene and displays an image in a visuallyrecognizable manner along with the outside scene, and the camera 61,which performs image capturing along the directions of user's sightlines. The control section 140 of the head mounted display 100 includesthe image generation section 177, which generates a display image from acaptured image from the camera 61, and the image display control section176, which causes the image display unit 20 to display the display imagegenerated by the image generation section 177. As a result, a displayimage based on a captured image captured along the directions of user'ssight lines can be so displayed that the display image is superimposedon a target object visible as the outside scene. A novel method forusing an image displayed by the display apparatus can therefore beprovided in the form of an effective combination of an outside sceneoutside the display apparatus and a displayed content, for example, byenhancing the way the user views the outside scene based on a capturedimage generated by capturing the outside scene or presenting another wayof viewing the outside scene.

The head mounted display 100 further includes the distance detectionsection 173 or any other detection section that detects the state inwhich the head mounted display 100 is used. The image display controlsection 176 adjusts the form in accordance with which the image displayunit 20 displays a display image in correspondence with the state inwhich the head mounted display 100 is used. As a result, the displaysize, the display position, and other aspects of an image displayed inthe image display unit 20 can be adjusted in correspondence with theenvironment in which the head mounted display 100 is used and the way inwhich the user uses the head mounted display 100, whereby the displayoperation can be performed in accordance with the way in which the uservisually recognizes an outside scene.

The distance detection section 173 or any other detection sectiondetects the position of the head mounted display 100, and the imagedisplay control section 176 adjusts the form in accordance of which thedisplay image is displayed in correspondence with the detected positionof the head mounted display 100, whereby the display operation can beperformed based on the position of the head mounted display 100 inaccordance with the way in which the user visually recognizes theoutside scene. Specifically, the image display control section 176adjusts at least one of the display position and the display size of thedisplay image in such a way that the display image is separate from thecenter of the directions of user's sight lines. The center of thedirections of user's sight lines refers, for example, not only to anintersection of a central axis of the sight lines in a case where theuser views an outside scene through the image display unit 20 and theentire display area where the image display unit 20 displays an imagebut also to a portion in the vicinity of the intersection. In this case,the display image is not displayed in a position that directly facesuser's sight lines, but the user visually recognizes the display imagein a position that does not directly face the sight lines (center ofsight lines). Therefore, according to the embodiment of the invention,an image captured with the camera 61 can be so displayed that the imagedoes not bother the user who views an outside scene. For example, thedisplay image can be so displayed that it does not hide a target objectthat the user desires to view.

Further, since the image generation section 177 cuts out and enlargespart of a captured image from the camera 61 to generate a display image,the captured image captured along user's sight lines can be displayed asan enlarged image. The user can therefore view an outside scene and aportion thereof along the directions of the sight lines as an enlargedimage.

Moreover, since the image generation section 177 cuts out an area thatsatisfies a preset condition in a captured image from the camera 61 togenerate a display image, a captured image that satisfies the presetcondition can be displayed. For example, when a captured image thatsatisfies user's preference or interest is captured, the image displayunit 20 can display the captured image and show it to the user.

The head mounted display 100 further includes the target detectionsection 171, which detects a target object in the directions of user'ssight lines from a captured image from the camera 61. The imagegeneration section 177 cuts out an area containing an image of thetarget object detected by the target detection section 171 from thecaptured image from the camera 61 to generate a display image.Therefore, when a target object is present in the directions of user'ssight lines, an image of the target object can be presented to the user.

The head mounted display 100 further includes the distance detectionsection 173, which detects the distance between the target objectdetected by the target detection section 171 and the user. The imagegeneration section 177 enlarges part of a captured image from the camera61 in accordance with an enlargement factor corresponding to thedistance detected by the distance detection section 173 to generate adisplay image, whereby an image of the target object can be enlarged inaccordance with the enlargement factor corresponding to the distance tothe target object present in the directions of user's sight lines, andthe enlarged image can be presented to the user.

Further, the image generation section 177 acquires a plurality ofcaptured images captured with the camera 61 at different points of timeand identifies areas that satisfy a preset condition in the plurality ofcaptured images to generate display images, whereby a complicatedcondition can be set on the plurality of captured images, and imagesbased on captured images that satisfy the condition can be presented tothe user.

The image generation section 177 acquires a plurality of captured imagescaptured with the camera 61 at different points of time, evaluateswhether the differences between the plurality of captured images satisfya preset condition, and cuts out part of the acquired captured images togenerate display images when the differences satisfy the presetcondition. Therefore, since display images can be generated anddisplayed based on the differences between the plurality of capturedimages, motion images formed of a plurality of continuous images can,for example, be used to generate display images based on changes in theimages and display the display images.

The image display control section 176 stores the display imagesgenerated by the image generation section 177 in the storage section 120in a time sequential manner and reproduces and displays the displayimages stored in the storage section 120 when the operation section 135detects operation that instructs reproduction of the display images. Theimage display control section 176 can display motion images bysequentially reproducing the plurality of display images stored in atime sequential manner at a predetermined frame rate. In this case, theuser can view motion images formed of recorded imaged enlarged aroundthe directions of the sight lines. In this process, before reproducingand displaying the display images stored in the storage section 120, theimage display control section 176 may change the enlargement factor inaccordance with which the display images have been enlarged. Forexample, when the operation section 135 detects operation of specifyingthe enlargement factor, the enlargement factor may be changed inaccordance with the operation. Further, in FIG. 5D, the display image isso displayed that it is superimposed on part of the field of view VR.However, when display images stored in the storage section 120 arereproduced and displayed, the display images may be displayed over theentire displayable areas of the image display unit 20.

Further, the image generation section 177 may adjust the transparency ofthe image IV. In the example shown in FIG. 5D, the transparency is zeroand no outside scene is therefore visible through the portion on whichthe image IV is superimposed. The image generation section 177 mayadjust the transparency based on the brightness of the outside scenesuperimposed on the image IV, the luminance of the image IV, and otherfactors.

In the embodiment described above, the description has been made withreference to the case where the target detection section 171 recognizesa target object by using the single image feature data 124 a or themotion image feature data 124 b stored in the storage section 120. Thesingle image feature data 124 a, which is stored in the storage section120 in advance, may also be generated or changed in accordance withuser's operation during the use of the head mounted display 100. Forexample, it is conceivable to configure the head mounted display 100 insuch a way that during the display operation in which the image displayunit 20 displays a captured image from the camera 61, part of thedisplay image can be selected in accordance with operation of thetrackpad 14. In this configuration, when the user selects part of thedisplay image, the target detection section 171 may detect an image of aperson or an object from the selected portion and generate single imagefeature data 124 a or motion image feature data 124 b from the detectedimage. The head mounted display 100 may instead be operated in responseto a voice command. In this case, the control section 140 detects user'svoice with the microphone 63, and the voice processing section 170analyzes the detected voice to recognize the voice command. During theoperation in which the head mounted display 100 causes the image displayunit 20 to display a captured image from the camera 61, when the userissues an instruction in the form of a voice command to select part of adisplay image, the target detection section 171 may detect an image of aperson or an object from the selected portion and generate single imagefeature data 124 a or motion image feature data 124 b from the detectedimage. Specifically, when an image of a football game is captured anddisplayed, as shown in FIG. 5A, and the user issues an instruction inthe form of a voice command to “track the player having a uniform numberof 11,” the target detection section 171 may detect the numeral “11”from a captured image from the camera 61, extract a portion containingthe detected numeral, and determine the amount of feature to generatesingle image feature data 124 a or motion image feature data 124 b.

Second Embodiment

FIG. 8 is a block diagram showing a functional configuration of a headmounted display 100B according to a second embodiment. The head mounteddisplay 100B has a configuration in which the image display unit 20 isreplaced with an image display unit 20B in the head mounted display 100described in the first embodiment. In the second embodiment, theportions configured in the same manner as in the first embodiment havethe same reference characters and will not be described. FIGS. 9A and 9Bshow the configuration of key portions of the image display unit 20B.FIG. 9A is a perspective view of the key portions of the image displayunit 20B viewed from the side where user's head is present, and FIG. 9Bdescribes the directions of sight lines.

The image display unit 20B includes distance sensors 64 and sight linesensors 68 in addition to the configuration of the image display unit 20(FIG. 2). The head mounted display 100 and the head mounted display 100Bare the same in terms of the other points, and the head mounted display100B can perform the action of the head mounted display 100 in the firstembodiment described above.

Reference characters 261A and 262A shown in FIG. 9A denote half-silveredmirrors formed on the right light guide plate 261 and the left lightguide plate 262, respectively, and correspond to the display areas wherean image is displayed and which can transmit an outside scene. The imagedisplay unit 20 in the first embodiment described above are alsoprovided with the half-silvered mirrors 261A and 262A.

The distance sensors 64 are disposed in a portion along the boundarybetween the right optical image display section 26 and the left opticalimage display section 28, as shown in FIG. 9A. In a state in which theuser wears the image display unit 20B, the positions of the distancesensors 64 are roughly at the middle between user's eyes in thehorizontal direction but above user's eyes in the vertical direction.The distance sensors 64 detect the distance to a target object undermeasurement located in a preset measurement direction.

For example, the distance sensors 64 detect the distance to a targetobject OB, which is located at an intersection of a sight line RD ofuser's right eye RE and a sight line LD of user's left eye LE, as shownin FIG. 9B. In this case, a direction 64A, in which the distance sensors64 performs the distance detection, is the frontward direction withrespect to the image display unit 20B.

Each of the distance sensors 64 includes, for example, a light source,such as an LED and a laser diode, and a light receiver that receiveslight emitted from the light source and reflected off a target objectunder measurement. In this case, the distance sensors 64 performdistance measurement based on triangulation or time difference under thecontrol of the distance detection section 173. Each of the distancesensors 64 may instead include a sound source that emits an ultrasonicwave and a detector that detects the ultrasonic wave reflected off atarget object under measurement. In this case, the distance sensors 64perform distance measurement based on the difference in time spent untilthe ultrasonic wave is reflected under the control of a positiondetection section 162, which will be described later. Instead, each ofthe distance sensors 64 may instead include a light source and a lightreceiver or a sound source and a detector, and the distance detectionsection 173 may perform the distance measurement.

The measurement direction of the distance sensors 64 in the presentembodiment is the frontward direction with respect to the head mounteddisplay 100B and coincides with the image capturing direction of thecamera 61.

The sight line sensors 68 are disposed on the user-side surface of theimage display unit 20B, as shown in FIG. 9A. The sight line sensors 68are provided as a pair in a central portion between the right opticalimage display section 26 and the left optical image display section 28in such a way that the sight line sensors 68 correspond to user's righteye RE (FIG. 9B) and user's left eye LE (FIG. 9B). The sight linesensors 68 are formed, for example, of a pair of cameras that captureimages of user's right eye RE and left eye LE. The sight line sensors 68perform image capturing under the control of the control section 140(FIG. 8), and the control section 140 detects light reflected off theeyeball surfaces of the right eye RE and the left eye LE or images ofthe pupils thereof from captured image data to identify the directionsof the sight lines. Further, the sight line sensors 68, which captureimages of user's right and left eyes, can detect, when the user closedthe eyes, that the eyes have been closed.

The distance sensors 64 and the sight line sensors 68 are connected tothe control section 140 via the connection section 40, as shown in theblock diagram of FIG. 8. The distance detection section 173 can controlthe distance sensors 64 to cause them to detect the distance to a targetobject. The distance detection section 173 may instead detect thedistance to a target object based on captured image data from the camera61, as described in the above first embodiment.

The head mounted display 100B according to the second embodiment cancapture action of any of user's hands, legs, eyes, and other portionswith the camera 61 and can be operated based on the actions of the user.The camera 61 can capture images of user's hands, and the sight linesensors 68 can capture images of user's eyes and detect motion thereof.Further, when the camera 61 is a wide-angle camera or a leg camera (notshown) facing downward with respect to the image display unit 20B isprovided, images of user's feet can be captured.

The image display control section 176 detects user's motion based oncaptured image data from the camera 61, captured image data from the legcamera, or detection results from the sight line sensors 68. Further,the image display control section 176, when the detected user's motionconforms to preset motion, detects the motion as operation.

In this case, the image display control section 176 functions as anaction detection section, the image display control section, and a voicedetection section.

User's motion to be detected as operation by the image display controlsection 176 is contained in the detection feature data 124 stored in thestorage section 120. The detection feature data 124 contains data on theamount of feature for extracting images of user's hands and legs from acaptured image from the camera 61, data on the amount of feature ofmotion of the extracted images of user's hands and legs, and other typesof data.

The image display control section 176 may instead be configured to becapable of detecting user's speaking action. In this case, the voiceprocessing section 170 converts voice collected with the microphone 63into digital voice data, and the image display control section 176processes the digital voice data. That is, the image display controlsection 176 generates the amount of feature of the voice from thedigital voice data that lasts for a predetermined time length, evaluateswhether or not the amount of feature conforms to the amount of featurefor voice recognition contained in the detection feature data 124, anddetects voice operation based on a result of the evaluation. In thisprocess, the image display control section 176 may perform voicerecognition on the digital voice data to generate text data on thespoken content and detect operation when the text data contains a presetcharacter string. In this case, the detection feature data 124 maycontain dictionary data for the voice recognition and data on thecharacter string to be detected as operation.

Action of the image display control section 176 that corresponds touser's action will be described with reference to FIGS. 10 and 11A to11D. FIG. 10 is a flowchart showing action of the head mounted display100B, and FIGS. 11A to 11D show display examples in the image displayunit 20B.

The action shown in FIG. 10 can be performed at an arbitrary point oftime after the display operation starts in step S23 in FIG. 4 and when apreset condition is satisfied or in response to user's operationperformed on the control unit 10.

The control section 140 transitions to a “user operation mode” that isan action mode for action based on user's motion and initiates detectionof user's motion (step S51). The motion detection is performed based oncaptured image data from the camera 61, data on detection results fromthe sight line sensors 68, or digital voice data generated by the voiceprocessing section 170, as described above.

The image display control section 176 evaluates whether or not thedetected user's motion conforms to motion set as “first motion” inadvance (step S52). The first motion is, for example, user's action ofclosing (shutting) one of the eyes, user's action of keeping legstogether, and user's action of speaking “start zooming.” The firstaction may instead be user's action of forming a circle with a thumb(first finger) and a forefinger (index finger) or a middle finger. Stillinstead, the first action may be user's action of facing the hands eachother and placing them within the image capturing rage of the camera 61.

Until action that conforms to the first action is detected (step S52;NO), the image display control section 176 waits for operation. Whenaction that conforms to the first action is detected (step S52; YES),the image display control section 176 initiates zooming adjustment (stepS53). In the zooming adjustment, an indicator is first displayed in theimage display unit 20B based on the function of the image displaycontrol section 176 (step S54). The indicator contains characters orimages showing that the user can perform zooming operation and is sodisplayed that the user can visually recognize the indicatorsuperimposed on an outside scene. The indicator may contain charactersor images that guide or describe operation that the user needs toperform to instruct zooming adjustment.

The image display control section 176 identifies a marker position thatserves as the center of the zooming adjustment and displays a marker inthe identified position (step S55). The marker is an image or a symbolshowing the user the center of the zooming adjustment and is displayedby the image display unit 20B.

As a method for identifying the marker position, there is, for example,a method used when the user closes one of the eyes as the first actionand including detecting the sight line of the unclosed eye and thedirection of the sight line is used to set the marker position. In thismethod, when user's eyes are both open, no single marker position ispossibly identified because the directions of the sight lines do notnecessarily coincide with each other. However, user's right or left eyemay be set in advance as a dominant eye through setting operationperformed on the control unit 10, and the direction of the sight line ofthe thus set dominant eye may be used to set the marker position. Inthis case, the dominant eye may be automatically determined.

When the user performs the action of forming a circle with a thumb and aforefinger or a middle finger as the first action, the center of thecircle can be set to be the marker position. When the user performs theaction of facing the hands each other and placing them within the imagecapturing range of the camera 61 as the first action, the center betweenthe hands may be set to be the marker position.

When the image display control section 176 can detect a plurality oftarget objects present in the directions of user's sight lines from acaptured image from the camera 61 and detect features of the detectedtarget objects, the marker position may be identified by allowing theuser to speak a word that specifies a target object. That is, the usermay speak a feature of a person or an object that is one of the targetobjects contained in the captured image from the camera 61, and theimage display control section 176 may select the one target object basedon the spoken word and identify the target object as the markerposition. In this case, the marker position can be specified by allowingthe user to speak a word that identifies an object, a place, or aperson, such as “ball,” “goal,” or “number 10.”

FIG. 11A shows a display example in a state in which a marker has beenrecognized. In FIG. 11A, the image display unit 20B displays anindicator IND, and the user forms a circle with fingers. In response tothe circle forming action of the user, the center of the circle isidentified as the marker position, and a marker M is displayed. The uservisually recognizes the indicator IND and the marker M in the field ofview of the user and views user's hand H as an outside scene.

The image display control section 176 initiates action of tracking thetarget object in the identified marker position, moving the imageaccordingly, and keeping the marker position located at the center ofthe display image (step S56).

The image display control section 176 then detects second action ofspecifying the zoom magnification or specifying a change in the zoommagnification (step S57). The second action can, for example, be user'saction of forming a circle with a thumb and a forefinger or a middlefinger and rotating the hand with the circle maintained. In this case,when the hand is rotated in the clockwise direction, the image displaycontrol section 176 increases the zoom magnification, whereas when thehand is rotated in the counterclockwise direction, the image displaycontrol section 176 decreases the zoom magnification. A change in thezoom magnification can be a unit amount set in advance. In this case,the user can adjust the zoom magnification to a desired value byrepeating the second action.

In the example shown in FIG. 11A, the user places the right hand withinthe image capturing range of the camera 61 and forms a circle withfingers. In this state, when the user moves the hand in such a way thatthe circle formed by the right hand is rotated in the clockwisedirection (CW in FIG. 11A), the zoom magnification increases in theimage IV, the center of which is the marker M, as shown in FIG. 11B.That is, the image IV is enlarged around the marker M, and the enlargedimage IV is displayed. Instead, when the user in the state shown in FIG.11A moves the right hand in such a way that the circle formed by theright hand is rotated in the counterclockwise direction (CCW in FIG.11A), the zoom magnification decreases, as shown in FIG. 11C. In FIG.11C, the image IV is visually recognized in the form of an image reducedaround the marker M.

When the user forms no circle but places one hand within the imagecapturing range of the camera 61, user's action of bringing the handtoward the camera 61 can be identified as the second actioncorresponding to an increase in the zoom magnification. Further, user'saction of bringing the hand away from the camera 61 can be identified asthe second action corresponding to a decrease in the zoom magnification.

Further, for example, when the user performs the action of facing thehands each other as the first action, the center between the hands maybe set to be the marker M, as shown in FIG. 11D. In this case, user'saction of increasing or decreasing the distance between the hands, asindicated by the arrows in FIG. 11D, can be identified as the secondaction. In this case, the image display control section 176 increasesthe zoom magnification when the user increases the distance between thehands, whereas decreasing the zoom magnification when the user decreasesthe distance between the hands.

The target detection section 171 detects an object corresponding to anyof user's limbs from a captured image from the camera 61 and identifiesthe position of the marker M based on the motion or the position of thedetected object, as illustrated in FIGS. 11A to 11D. The object to bedetected is not limited to a hand, an arm, and a leg of the user and maybe part of limbs, such as a finger and a palm.

Specifically, a thumb and a forefinger can be placed within the imagecapturing range of the camera 61, and the marker M can be displayed in aposition between the thumb and the forefinger. Further, the action ofdecreasing the distance between the thumb and the forefinger may beassigned to an instruction to decrease the zoom magnification, and theaction of increasing the distance between the thumb and the forefingermay be assigned to an instruction to increase the zoom magnification.Any of the other fingers may be used to perform the operation describedabove, or three or more fingers and motion thereof may be detected. Forexample, three or more fingers may be detected simultaneously, andmotion of each of the fingers, a change in relative positionalrelationship among the plurality of fingers, or any other behavior maybe detected as the motion. The process corresponding to the motion of afinger is not limited to the identification of the marker M and a changein the zoom magnification and may be volume adjustment of outputtedvoice or light control adjustment of a displayed image.

The second action may be user's action of moving a leg. User's action ofputting the right leg ahead of the left leg can be identified as thesecond action corresponding to an increase in the zoom magnification,and the zoom magnification may be increased in accordance with thelength of the period during which the right leg is put ahead of the leftleg. In this case, user's action of putting the right leg behind theleft leg can be identified as the second action corresponding to adecrease in the zoom magnification, and the zoom magnification may bedecreased in accordance with the length of the period during which theright leg is put behind the left leg. The zoom magnification may insteadbe changed in accordance with the amount of the right leg put ahead ofor behind the left leg. In this case, the indicator may display markingsshowing a change in the zoom magnification and the correspondence of thezoom magnification to the amount of one leg put ahead of or behind theother leg.

Further, the second action may be user's spoken voice. The user mayspeak a word that specifies zoom-up, zoom-down, or the zoommagnification in the form, for example, of a numeral, and the imagedisplay control section 176 may adjust the zoom magnification.

Unless action that conforms to the second action is detected (step S57;NO), the image display control section 176 waits for user's operation.When action that conform to the second action is detected (step S57;YES), the image display control section 176 changes the zoommagnification (step S58). The image display control section 176 thendetects third action.

The third action corresponds to operation of issuing an instruction toinitiate recording. The third action can, for example, be user's actionof placing one hand within the image capturing range of the camera 61and inclining the hand, user's action of putting the left leg ahead ofthe right leg, or user's action of closing both eyes. The third actionmay instead be user's action of speaking a word that instructs toinitiate recording. When the third action is detected (step S59; YES),the image display control section 176 initiates recording of capturedimages from the camera 61 and digital voice data on voice collected withthe microphone 63 and generated by the voice processing section 170(step S60).

The image display control section 176 then detects fourth action (stepS61). The fourth action corresponds to operation of issuing aninstruction to stop the recording. For example, when the user hasperformed action of placing one hand within the image capturing range ofthe camera 61 and inclining the hand as the third action, the fourthaction can be action of moving the inclined hand back to the originalposition. When the user has put the left leg ahead of the right leg asthe third action, the fourth action can be action of moving the left legbackward. The fourth action can instead be user's action of closing botheyes. The fourth action can still instead be user's action of speaking aword that instructs to stop the recording. When the fourth action isdetected (step S61; YES), the image display control section 176 stopsthe recording (step S62).

The image display control section 176 then evaluates whether or not thezooming adjustment operation is terminated (step S63). The image displaycontrol section 176 performs the evaluation in step S63 also when nothird action is detected in a predetermined period in step S59 (stepS59; NO).

When user's operation performed on the operation section 135, or a hand,a leg, an eye (eyelid), voice, or any other type of action as in thefirst action to the fourth action instructs to stop the zoomingoperation (step S63; YES), the image display control section 176terminates the user operation mode, stops displaying the indicator,transitions to a normal display mode (step S64), and terminates theentire process. When a result of the evaluation in step S63 shows thatthe user does not desire to terminate the zoom operation (step S63; NO),the image display control section 176 returns to step S57.

As described above, the head mounted display 100B can be operated inresponse to user's action of moving a hand, a leg, or an eye (eyelid),speaking a word, or performing any other type of behavior. A variety oftypes of instruction operation can therefore be performed without directoperation of the control unit 10, whereby the head mounted display 100Bcan be operated, for example, with the control unit 10 and the imagedisplay unit 20B worn on the body or the clothing and the hands keptfree or in what is called a hands-free state. Further, since the headmounted display 100B can be operated in response to eye, voice, or legaction, the head mounted display 100B can be operated even when thehands are used in other types of activity and hence the user cannotmanually operate the head mounted display 100B.

Further, in steps S59 and S61, when a hand or a leg is placed within theimage capturing range of the camera 61 as the third action and thefourth action, the image display control section 176 may carry out aprocess of preventing the hand or the leg from being contained inrecorded images. That is, the image display control section 176, when itdetects hand or leg action as the third action and the fourth action,may carry out a process of removing an image of the hand or the legdetected as the third action and the fourth action from images to berecorded. For example, the image of the hand or the leg can be removedby trimming frames of captured images from the camera 61.

Further, before the action shown in FIG. 10, the control section 140 mayperform calibration that relates the directions of the sight linesdetected by the sight line sensors 68 to the image capturing range ofthe camera 61. A result of the calibration may be stored along, forexample, with the detection feature data 124 in the storage section 120.

Further, the action shown in FIG. 10 has been described with referenceto the case where the zoom magnification is adjusted in steps S53 to S58and the recording is then performed in steps S59 to S62, but only therecording may be performed. Moreover, in the first action detection stepof step S52, when action set in advance as the third action is detected,the action in step S60 may be performed.

Further, in addition to the hand and leg action, action associated withan indicating object other than a hand or a leg can be detected as thefirst action to the fourth action. In this case, the indicating objectonly needs to be an object having a feature that can be extracted by theimage display control section 176 from a captured image from the camera61 and may be an object having a specific shape, color, or pattern.Further, an indicating object can be a device that emits light that doesnot fall within a visible region, such as infrared light and ultravioletlight, as long as the camera 61 can receive and detect light of thistype. Operation of moving any of the indicating objects described abovealong a predetermined trajectory within the image capturing range of thecamera 61 or placing any of the indicating objects within the imagecapturing range of the camera 61 can be identified as any of the firstaction to the fourth action. When any of the indicating objects is awearable device that can be worn on user's body or clothing, such as awristwatch, operability can be further improved.

Further, the first action to the fourth action or the action listedabove as the action for identifying the marker position as describedabove can not only be used to instruct the adjustment of the zoommagnification and the recording but also be applied, for example, tooperation of reproduction of recorded images.

FIG. 12 shows a display example in a case where an instruction toreproduce recorded motion images is issued. The display example shown inFIG. 12 is displayed, for example, when an instruction of thereproduction is issued in response to operation performed on the controlunit 10. In FIG. 12, a timeline TL showing entire recorded motion imagesis displayed in the field of view VR. For the frames at specific pointsof time along the timeline TL, the image display control section 176generates and displays thumbnail images P1 to P3. The point of time atwhich a thumbnail image is generated and displayed may be specifiedduring recording by user's specific action, such as any of the firstaction to the fourth action or may be automatically selected by theimage display control section 176 in accordance with the temporal lengthof recorded images. The image display control section 176 may insteadautomatically select the point of time based on a change in images. Inthe state in which the display example shown in FIG. 12 is displayed,the time at which reproduction is initiated can be selected from thethumbnail images P1 to P3 based, for example, on the directions of thesight lines.

As described above, according to the second embodiment to which theinvention is applied, the image display control section 176 detectsuser's action based on a captured image from the camera 61 and adjuststhe form in accordance with which a display image is displayed incorrespondence with the detected user's action. For example, the zoommagnification is adjusted. Therefore, by allowing the user to performaction an image of which can be captured with the camera 61, the form inaccordance with which a display image is displayed can be adjusted.Instead, the image display control section 176 may detect voicecollected with the microphone 63 and adjust the form in accordance withwhich a display image is displayed in correspondence with a result ofthe voice detection.

The image display control section 176 detects user's action based on acaptured image from the camera 61. The image display control section 176can then extract and process an area identified based on the user'saction, for example, an area around the marker M to cause the imagegeneration section 177 to generate a display image.

The image display control section 176 identifies directionscorresponding to the directions of user's sight lines and detects atarget object present in the directions of user's sight lines from acaptured image from the camera 61. The image generation section 177 maythen extract and process an area containing an image of the targetobject to generate a display image. Instead, when directionscorresponding to the directions of user's sight lines are identified,the marker M may be displayed in a position corresponding to theidentified directions.

Further, directions corresponding to the directions of user's sightlines may be determined by detecting the sight line of user's dominanteye. Moreover, motion of user's hand or leg may be detected from acaptured image from the camera 61, and the position of the marker M maybe identified based on the motion.

The invention is not limited to the configuration in each of theembodiments described above and can be implemented in a variety ofaspects to the extent that they do not depart from the substance of theinvention.

In the first embodiment described above, as an example of the operationin which the image generation section 177 cuts out an area thatsatisfies a preset condition from a captured image from the camera 61,the case where the image generation section 177 cuts out an areacontaining a target object detected in the matching process performed bythe target detection section 171 has been described. The invention isnot limited to the case described above. For example, the imagegeneration section 177 may detect and cut out an area that satisfies apreset condition in a captured image to generate a display image. Forexample, in a case where as the condition under which the imagegeneration section 177 cuts out an area from a captured image, theposition and size of the area are set in advance, the image generationsection 177 cuts out an area located in the set position and having theset size from a captured image irrespective of a detection result fromthe target detection section 171. Further, the condition under which anarea is cut out may instead be a condition that specifies the color orluminance in a captured image, and the target detection section 171 isnot necessarily required to perform some type of processing.

Further, the process carried out when the image generation section 177generates a display image from a captured image is not limited to theprocess of cutting out and enlarging part of the captured image. Forexample, the entire captured image as it is or the captured image afterresolution conversion may be used as a display image. Moreover, forexample, the image generation section 177 may cut out a plurality ofareas from a captured image to generate a plurality of display imagesand display a plurality of images IV simultaneously in the field of viewVR. Further, the image generation section 177 or the image displaycontrol section 176 may change or adjust as appropriate the positionwhere the display image is displayed.

In the first and second embodiments described above, the image capturingdirection of the camera 61 may be movable relative to the image displayunits 20 and 20B. For example, the camera 61 may be attached to a frameof each of the image display units 20 and 20B via a rotatable arm orlink or a flexible arm so that the orientation of the camera 61 ischangeable without movement of the image display unit 20 or 20B. In thiscase, the angle of the optical axis or the image capturing direction ofthe camera 61 with respect to the image display units 20 and 20B may bedetectable. Specifically, a rotation sensor may be provided at a pivotalportion of the link or the arm that supports the camera 61, or thecamera 61 itself may be provided with a gyro sensor or an accelerationsensor. In this case, the directions of user's sight lines may bedetermined based on the direction of the optical axis of the camera 61.When the optical axis of the camera 61 is movable, since the user canorient the optical axis of the camera 61 in a desired direction, theorientation of the optical axis of the camera 61 can be considered asthe direction of user's attention or a direction associated therewith.The position of the head mounted display 100 can therefore bedetermined, for example, in step S31 in FIG. 6 based on the orientationof the optical axis of the camera 61. Further, the position of themarker can be identified in step S55 in FIG. 10 based on the orientationof the optical axis of the camera 61.

Further, the image display control section 176 may control the motion inimages displayed by the image display unit 20. For example, the imagedisplay control section 176 may control images displayed in the imagedisplay unit 20 based on the motion of the image display unit 20 thatcan be detected from detected values from the nine-axis sensor 66 and/orthe GPS 115. When the acceleration or speed of the motion of the imagedisplay unit 20 reaches or exceeds a threshold in the period duringwhich images generated from captured images from the camera 61 aredisplayed, the image display control section 176 may suppress the motionin the images displayed in the image display unit 20 or reverse themotion in the displayed images in the direction opposite the movement ofthe image display unit 20. In this case, the image display controlsection 176 may change display images generated by the image generationsection 177 or display part of the display images. According to theconfiguration described above, when the user's head around which thehead mounted display 100 is worn moves fast and displayed imagestherefore greatly change, the motion in images displayed by the imagedisplay unit 20 can be suppressed, whereby discomfort felt by the usercan be reduced. The function described above is called a stabilizerfunction. The stabilizer function can be switched between ON and OFF inresponse to operation performed on the operation section 135. Further,as described with reference, for example, to FIGS. 10 and 11A to 11D,the stabilizer function can be switched between ON and OFF in responseto motion of user's hand, leg, or eye or user's voice.

The stabilizer function described above is also applicable to a casewhere a target object detected by the target detection section 171 movesat a high speed in captured images. For example, when the ball isdetected as a target object, as illustrated in FIGS. 5A to 5C, themovement speed of the ball may be determined from changes in theposition of the ball in captured images, and changes in displayed imagesor the movement of the ball in a displayable position in the imagedisplay unit 20 may be suppressed when the acceleration or speed of theball is greater than or equal to a threshold. Instead, when the movementspeed or the acceleration determined from changes in the position of theball in captured images is greater than or equal to a threshold, theimage generation section 177 may temporarily stop the process ofextracting an image of the target object to generate a display image.Still instead, the image generation section 177 may lower theenlargement factor according to which an extracted image is enlarged,change the angle of view in the image capturing performed by the camera61, or lower the zoom magnification in a case where the camera 61 has azooming mechanism or a digital zoom function. As described above, it isadvantageously expected that reducing the display size of an image of atarget object displayed by the image display unit 20 allows the targetobject to be readily viewed even when the target object moves at a highmovement speed.

Further, for example, in the embodiments described above, thedescription has been made of the case where the image generation section177 determines the enlargement factor and the image generation section177 cuts out and enlarges part of a captured image by using imageprocessing in accordance with the enlargement factor. The invention isnot limited to the case. For example, the camera 61 may has a zoomfunction, and the image generation section 177 may control the zoommagnification of the camera 61. In this case, the image generationsection 177 may acquire a captured image captured at a zoommagnification specified by the image generation section 177 and cut outan image to generate a display image.

As the image display unit, the image display unit 20 may be replacedwith an image display unit worn, for example, as a cap or any otherimage display unit worn based on another method. A display unit thatdisplays an image in correspondence with user's left eye and a displayunit that displays an image in correspondence with user's right eye maybe provided. Moreover, the display apparatus according to any of theembodiments of the invention may, for example, be configured as a headmounted display incorporated in an automobile, an airplane, and othervehicles. Further, for example, the display apparatus may be configuredas a head mounted display built in a helmet or other body protectiongears or may be a head-up display (HUD) used with a windshield of anautomobile. Still further, the image display unit 20 may be a displaythat forms an image on the retina in each user's eyeball, such as whatis called a contact lens display attached to user's eyes (onto cornea,for example) when used and an implanted display embedded in eacheyeball.

The invention can be applicable to an apparatus worn on user's body andis applicable to such an apparatus irrespective of whether or not theapparatus needs to be supported by using another method. For example, abinocular-type handheld display used by a user who holds it with handsmay be employed as the image display unit 20 in the present application.Although a display of this type requires the user to hold it with thehands in order to maintain the apparatus attached to the user, theapparatus falls within the range of the display apparatus according tothe embodiments of the invention because the apparatus comes intocontact with user's head or face when the user views an image displayedin the display. Further, even a display apparatus fixed to a floorsurface or a wall surface, for example, by using support legs fallswithin the range of the display apparatus according to the embodimentsof the invention as long as the display apparatus comes into contactwith user's head or face when the user views an image displayed in thedisplay.

Further, the image display unit 20 or only a display unit including aconfiguration involved in image display in the image display unit 20 maybe worn on user's body, and the control unit 10 excluding the displayunit or a control system including the control unit 10 and the controlsection 140 may be physically separated. For example, the image displayunit 20 or a display unit formed of part of the image display unit 20may be wirelessly connected to an apparatus including another controlsystem to form a display apparatus, as in the case of the head mounteddisplay 100. Examples of the apparatus including the control system mayinclude a smartphone, a mobile phone, a tablet computer, a personalcomputer having a shape different from that of a tablet computer, andother existing electronic apparatus. The present application is, ofcourse, applicable to a display apparatus of this type.

Further, in the embodiments described above, the description has beenmade of the case where the image display units 20 and 20B are separatedfrom the control unit 10 and they are connected to each other via theconnection section 40. The control unit 10 and the image display unit 20can instead be integrated with each other, and the integrated unit canbe worn around user's head.

Further, the control unit 10 and the image display units 20 and 20B areconnected to each other via a longer cable or a wireless communicationline, and the control unit 10 may, for example, be a mobile electronicapparatus including a notebook computer, a tablet computer, a desktopcomputer, a game console, a mobile phone, a smartphone, and a mobilemedia player, or any other dedicated apparatus.

Further, for example, the configuration that generates image light inthe image display unit 20 may be a configuration including an organic EL(organic electro-luminescence) display and an organic EL controller.Moreover, an LCOS (liquid crystal on silicon) device (LCoS is aregistered trademark), a digital micromirror device, or any other devicecan be used as the image light generation configuration. Further, forexample, the invention is also applicable to alaser-retina-projection-type head mounted display. That is, aconfiguration in which the image generation section may include a laserlight source and an optical system that guides a laser beam to user'seyes may be employed. In this configuration, the laser beam may becaused to enter each user's eye, and the retina may be scanned with thelaser beam, so that the user is allowed to visually recognize an image.When a laser-retina-projection-type head mounted display is used, the“region of the image light generation unit through which image light isallowed to exit” can be defined as an image region recognized by user'seye.

As the optical system that guides image light to user's eyes, anemployable configuration includes an optical member that transmitsexternal light externally incident on the display apparatus and allowsthe external light along with image light to be incident on user's eyes.Another usable optical member may be disposed in a position in front ofuser's eyes and may coincide with part or the entirety of the field ofview of the user. Still another employable optical system may be ascan-type optical system that sweeps, for example, a laser beam to formimage light. The optical system does not necessarily guide image lightthrough an optical member and may only have a function of guiding imagelight toward user's eyes based on refraction and/or reflection.

The invention is also applicable to a display apparatus that employs ascan optical system using a MEMS mirror and uses a MEMS displaytechnology. That is, an image display device may include a signal lightformation section, a scan optical system having a MEMS mirror thatsweeps light outputted from the signal light formation section, and anoptical member on which the light swept by the scan optical system formsa virtual image. In this configuration, the light outputted from thesignal light formation section is reflected off the MEMS mirror, isincident on the optical member, is guided through the optical member,and reaches a virtual image formation plane. The MEMS mirror sweeps thelight to form a virtual image on the virtual image formation plane, andthe user captures the virtual image to recognize an image. The opticalmember in this case may be a multi-reflection light guide, such as theright light guide plate 261 and the left light guide plate 262 in theembodiments described above, or may be a half-silvered mirror.

The display apparatus according to any of the embodiments of theinvention is not limited to a head mounted display and is applicable toa flat panel display, a projector, and a variety of other displayapparatus. A display apparatus according to an embodiment of theinvention only needs to allow the user to visually recognize an imageformed by image light along with external light and may, for example,have a configuration in which an optical member that transmits externallight allows visual recognition of an image formed by image light.Specifically, in addition to the head mounted display described abovehaving the configuration in which an optical member that transmitsexternal light is provided, the invention is also applicable to adisplay apparatus that projects image light on a light transmissive flatsurface or a curved surface (made, for example, of glass or transparentplastic material) installed in a position separate from the user in animmobile or movable manner. A display apparatus of this type may, forexample, be so configured that image light is projected on a window paneof a vehicle and a user in the vehicle and a user outside the vehicleare allowed to visually recognize an image formed by the image lightalong with scenes inside and outside the vehicle. Another displayapparatus of this type may, for example, be so configured that imagelight is projected on a transparent, semi-transparent, or coloredtransparent display surface, such as a windowpane of a building,installed in an immobile manner and users around the display surfacevisually recognize an image formed by the image light along with a scenethrough the display surface.

At least part of the functional blocks shown in FIGS. 2 and 8 may beachieved by hardware or hardware and software cooperating with eachother, and the configuration formed of independent hardware resourcesshown in FIG. 2 is not necessarily employed.

The action of the control section 140 described in the above first andsecond embodiments can instead be achieved in the form of a program.That is, the control section 140 may include a CPU, a ROM, and a RAM,and the CPU may execute a program to achieve, for example, the functionsof the target detection section 171, the distance detection section 173,the image display control section 176, and the image generation section177. Further, the CPU in the control section 140 may execute a programto perform the functions of the operating system 150, the imageprocessing section 160, the voice processing section 170, and thedisplay control section 190. Part or the entirety of these programs maybe achieved by hardware. For example, each of the processing sectionsdescribed above provided in the control section 140 may be formed of anASIC (application specific integrated circuit), a programmed FPGA(field-programmable gate array) or any other programmed PLD(programmable logic device), or a circuit including a plurality ofsemiconductor devices. When the functions described above are achievedby hardware that executes a program, the program may be stored in theROM in the control section 140, the storage section 120, or anotherstorage device in the control unit 10. Instead, the control section 140may acquire a program stored in an external apparatus via thecommunication section 117 or the interface 180 and execute the program.

Among the configurations formed in the control unit 10, only theoperation section 135 may be formed as an independent user interface(UI), and the power supply 130 in the embodiments described above may beindependently formed in an exchangeable manner. Further, theconfigurations formed in the control unit 10 may be redundantly formedin the image display unit 20. For example, the control section 140 shownin FIG. 2 may be formed both in the control unit 10 and the imagedisplay unit 20, and the control section 140 formed in the control unit10 and the CPU formed in the image display unit 20 may perform differentfunctions.

The entire disclosure of Japanese Patent Application Nos. 2014-156697,filed Jul. 31, 2014 and 2015-039766, filed Mar. 2, 2015 are expresslyincorporated by reference herein.

What is claimed is:
 1. A head-mountable display apparatus that isconfigured to be able to be worn on a head of a user, the head-mountabledisplay apparatus comprising: a display that displays an image in avisually recognizable manner superimposed on outside real world scenery;a camera that performs image capturing along directions of sight linesof the user; and a processor programmed to: cause the camera to capturean image; detect, in the captured image, a target object; detect adistance between the detected target object and the head-mounted displayapparatus; generate a display image from the captured image by cuttingout, from the captured image, an area containing the detected targetobject, and enlarging, based on the detected distance, the cut out areacontaining the detected target object, wherein the generated displayimage is an enlarged version of the area containing the detected targetobject cut out of the captured image; and control the display to displaythe generated display image superimposed on the outside real worldscenery, wherein the display image and the outside real world sceneryare live view.
 2. The head-mountable display apparatus according toclaim 1, wherein the processor is further programmed to: detect a statein which the head-mountable display apparatus is used, and adjust a formof the display image in accordance with the detected state.
 3. Thehead-mountable display apparatus according to claim 1, wherein theprocessor is further programmed to: detect a position of thehead-mountable display apparatus, and adjust a form of the display imagein accordance with the detected position.
 4. The head-mountable displayapparatus according to claim 3, wherein the processor is furtherprogrammed to: adjust at least one of a display position of the displayimage and a display size of the display image in such a way that thedisplay image is displayed on the display separate from the center ofthe directions of the sight lines of the user in correspondence with thedetected position.
 5. The head-mountable display apparatus according toclaim 1, wherein the processor is further programmed to: detect anaction of the user based on captured image data from the camera, andadjust a form of the display image in accordance with the detectedaction of the user.
 6. The head-mountable display apparatus according toclaim 1, wherein the processor is further programmed to: detect a voice,and adjust a form of the display image in accordance with the detectedvoice.
 7. The head-mountable display apparatus according to claim 1,wherein the processor is further programmed to: detect an action of theuser based on captured image data from the camera, and generate thedisplay image by extracting, from the captured image, an area identifiedbased on the detected action of the user and processing the extractedarea.
 8. The head-mountable display apparatus according to claim 1,wherein the processor is further programmed to: generate the displayimage by cutting out an area that satisfies a preset condition in thecaptured image.
 9. The head-mountable display apparatus according toclaim 1, wherein the processor is further programmed to: identifydirections corresponding to the directions of the sight lines of theuser; and detect, in the captured image, a target object present in thedirections of the sight lines of the user, as the detected targetobject.
 10. The head-mountable display apparatus according to claim 9,wherein the processor is further programmed to: detect the directions ofthe sight lines of the user and detect the target object present in thedetected directions of the sight lines of the user.
 11. Thehead-mountable display apparatus according to claim 10, wherein theprocessor is further programmed to: detect a direction of a sight lineof an eye corresponding to a dominant eye of the user.
 12. Thehead-mountable display apparatus according to claim 7, wherein theprocessor is further programmed to: when the detected action of the useris a predetermined action, identify the directions corresponding to thedirections of the sight lines of the user based on the predeterminedaction.
 13. The head-mountable display apparatus according to claim 7,wherein the processor is further programmed to: when the detected actionis an action of at least one of limbs of the user, identify thedirections corresponding to the directions of the sight lines of theuser based on the detected action.
 14. The head-mountable displayapparatus according to claim 9, wherein the processor is furtherprogrammed to: store, in a memory, an amount of feature of the targetobject, and detect the target object by detecting an image that conformsto the amount of feature of the target object stored in the memory fromthe captured image.
 15. The head-mountable display apparatus accordingto claim 9, wherein the processor is further programmed to: generate thedisplay image by enlarging the area containing the detected targetobject in accordance with an enlargement factor associated with thedetected distance.
 16. The head-mountable display apparatus according toclaim 1, wherein the processor is further programmed to: generate thedisplay image by acquiring a plurality of images captured by the cameraat different points of time and specifying an area that satisfies apreset condition in the plurality of captured images.
 17. A method forcontrolling a head-mountable display apparatus that is configured to beable to be worn on a head of a user, the method comprising: performingimage capturing, by a camera of the head-mountable display apparatus,along directions of sight lines of a user; detecting, by a processor ofthe head-mounted display apparatus, a target object in a captured image;detecting, by the processor, a distance between the detected targetobject and the head-mounted display apparatus; generating, by theprocessor, a display image from the captured image by cutting out anarea, from the captured image, containing the detected target object,and enlarging, based on the detected distance, the cut out areacontaining the detected target object, wherein the generated displayimage is an enlarged version of the area containing the detected targetobject cut out of the captured image; and controlling, by the processor,the display to display the generated display image superimposed onoutside real world scenery, wherein the display image and the outsidereal world scenery are live view.
 18. A non-transitory computer readablemedium that stores a computer-executable program that, when executed bya head-mounted display device, causes the head-mounted display deviceto: cause a camera of the head-mountable display apparatus to capture animage along directions of sight lines of a user; detect, in the capturedimage, a target object; detect a distance between the detected targetobject and the head-mounted display apparatus; generate a display imagefrom the captured image by cutting out, from the captured image, an areacontaining the detected target object, and enlarging, based on thedetected distance, the cut out area containing the detected targetobject, wherein the generated display image is an enlarged version ofthe area containing the detected target object cut out of the capturedimage; and control a display of the head-mountable display apparatus todisplay the generated display image superimposed on outside real worldscenery, wherein the display image and the outside real world sceneryare live view.
 19. The head-mountable display apparatus according toclaim 1, wherein the detected target object is a target object thatsatisfies a predetermined condition under which the processor cuts outthe area containing the detected target object.
 20. The head-mountabledisplay apparatus according to claim 1, wherein the processor is furtherprogrammed to: determine a size of the area cut out from the capturedimage based on a display size, which is a resolution or number of imagedots, in accordance with which the display image is displayed indisplayable areas of the display and an associated enlargement factor;and perform the enlargement step by converting the size of the area cutout from the captured image in accordance with the display size.
 21. Thehead-mountable display apparatus according to claim 1, furthercomprising a memory, wherein the memory is configured to store:enlargement factor setting data, which relates distances to detectedtarget objects to enlargement factors.
 22. The head-mountable displayapparatus according to claim 1, wherein the processor is furtherconfigured to: not allow generation or display of the display image whenthe distance to the detected target object is shorter than a presetdistance.
 23. The head-mountable display apparatus according to claim 1,wherein the processor is further programmed to: detect a direction of asight line of the user; and detect the target object present in thedetected direction of the sight line of the user.
 24. The head-mountabledisplay apparatus according to claim 1, wherein the processor is furtherprogrammed to: detect a direction of a sight line of the user based on adetected sight line of an eye corresponding to a dominant eye of eyes ofthe user.